Physical information acquisition method, physical information acquisition device, and semiconductor device

ABSTRACT

A physical information acquisition method in which a corresponding wavelength region of visible light with at least one visible light detection unit coupled to an image signal processing unit is detected, each said visible light detection unit comprising a color filter adapted to transmit the corresponding wavelength region of visible light; a wavelength region of infrared light with at least one infrared light detection unit coupled to the image signal processing unit is detected; and, with the signal processing unit, a first signal received from the at least one visible light detection unit by subtracting a product from said first signal is corrected, said product resulting from multiplication of a second signal received from the at least one infrared light detection unit and a predetermined coefficient factor.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.12/604,891, filed Oct. 23, 2009, which is a continuation of U.S. patentapplication Ser. No. 11/458,871, filed Jul. 20, 2006, the entirety ofwhich is incorporated herein by reference, and claims the prioritybenefit of Japanese Patent Application JP 2005-211002, filed on Jul. 21,2005, and Japanese Patent Application JP 2006-133412, filed on May 12,2005, the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a physical information acquisitionmethod, a physical information acquisition device, and a semiconductordevice. In more detail, the present invention relates to a signalacquisition technique appropriate for applying to a solid imagecapturing apparatus and so forth which use a semiconductor device forphysical quantity distribution detection which is configured of multipleunit components, having sensitivity as to electromagnetic waves to beinput from the outside such as light, radiation, or the like, beingarrayed, and can read out a physical quantity distribution convertedinto an electric signal by the unit components as an electric signal.Particularly, the present invention relates to an arrangement foreliminating influence of unnecessary wavelength (a typical example isinfrared light and ultraviolet light as to visible light) regioncomponents which filter in a principal wavelength region (a typicalexample is a visible light region).

2. Description of the Related Art

Physical quantity distribution detecting semiconductor devices, whichare configured of a plurality of unit components (e.g., pixels) havingsensitivity as to change in physical quantity such as electromagneticwaves to be input from the outside such as light, radiation, or the likebeing arrayed in a line shape, or in a matrix form, are employed invarious fields.

For example, with the visual equipment field, a CCD (Charge CoupledDevice) type for detecting change in light serving as one example ofphysical quantity (one example of electromagnetic waves) or MOS (MetalOxide Semiconductor), or CMOS (Complementary Metal-oxide Semiconductor)type solid image capturing apparatuses are employed. These read out aphysical quantity distribution converted into an electric signal by unitcomponents (pixels in the case of solid image capturing apparatuses) asan electric signal.

For example, a solid state image capturing device detectselectromagnetic waves to be input from the outside such as light,radiation, and so forth using a photodiode serving as an photoelectricconversion device (light receiving device; photo sensor) provided in theimage capturing unit (pixel unit) of a device unit to generate andaccumulate signal electric charge, and reads out the accumulated signalelectric charge (photoelectron) as image information.

Also, recently, arrangements for capturing an image using visible lightand capturing an image using infrared light have been proposed (e.g.,see U.S. Pat. No. 5,965,875, Japanese Unexamined Patent ApplicationPublication No. 2004-103964, Japanese Unexamined Patent ApplicationPublication No. 10-210486, Japanese Unexamined Patent ApplicationPublication No. 2002-369049, Japanese Unexamined Patent ApplicationPublication No. 06-121325, Japanese Unexamined Patent ApplicationPublication No. 09-166493, Japanese Unexamined Patent ApplicationPublication No. 09-130678, Japanese Unexamined Patent ApplicationPublication No. 2000-59798, and Japanese Unexamined Patent ApplicationPublication No. 2003-70009). For example, the position of the emittinglight point of infrared rays is prepared beforehand to trace this,whereby the position of the emitting light point of infrared lightpresent within a visible light image can be detected. Also, even in theevent of no visible light, e.g., even in the event of night, a brilliantimage can be obtained by illuminating infrared light to capture animage. Further, in addition to visible light, infrared light isemployed, whereby sensitivity can be improved.

The arrangements described in U.S. Pat. No. 5,965,875, and JapaneseUnexamined Patent Application Publication No. 2004-103964 relate to asingle-plate type utilizing the difference of absorption coefficientsbased on wavelengths in the depth direction of a semiconductor.

Also, the arrangements described in Japanese Unexamined PatentApplication Publication No. 10-210486, Japanese Unexamined PatentApplication Publication No. 2002-369049, and Japanese Unexamined PatentApplication Publication No. 06-121325 relate to a multi-plate type forreceiving visible light and infrared light at separate image capturingdevices using a wavelength resolution optical system such as awavelength separating mirror or prism or the like as an input opticalsystem.

Also, the arrangement described in Japanese Unexamined PatentApplication Publication No. 09-166493 relates to a single-plate type forreceiving visible light and infrared light at the same image capturingdevice using a rotating type wavelength resolution optical system as aninput optical system. For example, insertion/extraction of an infraredlight cut filter is performed in a rotating-mechanical manner, and wheninserting an infrared cut filter, a visible color image which has noinfluence from near-infrared light and infrared light is output, butwhen extracting an infrared light cut filter, an image to which thelight intensity of visible light and near-infrared light is added isoutput.

Also, the arrangement described in Japanese Unexamined PatentApplication Publication No. 09-130678 relates to a type for receivingvisible light and infrared light at the same image capturing deviceusing a diaphragm optical system having a wavelength resolution functionas an input optical system.

Also, the arrangement described in Japanese Unexamined PatentApplication Publication No. 2000-59798 is an arrangement wherein a colorfilter for transmitting near-infrared light is disposed on an imagecapturing device having sensitivity as to near-infrared light andvisible light, and also adjustment means are provided for adjusting theposition of the infrared cut filter between a position where incidentlight to the image capturing device passes through the infrared cutfilter and a position where incident light to the image capturing devicedoes not pass through the infrared cut filter, and when sharing theinfrared cut filter for photographing using near-infrared light and forphotographing using visible light, both of the sensitivity as to anear-infrared light region and the sensitivity as to a visible region ofthe image capturing device are effectively utilized by switching theposition of the infrared cut filter.

Also, the arrangement described in Japanese Unexamined PatentApplication Publication No. 2003-70009 is an arrangement whereincorrection for reducing the value of a color difference signal and/orluminance signal depending on the influence degree of infrared light asto the value of a color difference signal or luminance signal toeliminate the influence of infrared light. As one example, anarrangement is made wherein in the event that color filters arecomplementary-color filters of magenta, green, cyan, and yellow,correction is performed as to a color difference signal R-Y, and eachcolor output signal of magenta and yellow.

SUMMARY OF THE INVENTION

FIGS. 1A and 1B are diagrams for describing the arrangement of thesensors described in U.S. Pat. No. 5,965,875 and Japanese UnexaminedPatent Application Publication No. 2004-103964, wherein FIG. 1A is adiagram illustrating the photoabsorption spectrum properties ofsemiconductor layers, and FIG. 1B is a schematic view of thecross-sectional configuration of a device.

According to this arrangement, the photoabsorption coefficient of a Si(Silicon) semiconductor becomes small in the sequence of blue, green,red, and infrared light, as illustrated in FIG. 1A, i.e., with regard toblue light, green light, red light, and infrared light included in anincident light L1, an advantage wherein a wavelength exhibits placedependency in the depth direction of a semiconductor is utilized, layersfor detecting each color light of visible light (blue, green, and red)and infrared light are provided in order in the depth direction from thesurface of the Si semiconductor, as illustrated in FIG. 1B.

However, with the arrangement described in Japanese Unexamined PatentApplication Publication No. 2004-103964 utilizing the difference ofabsorption coefficients according to a wavelength, the amount of lightwhich can be detected logically is not deteriorated, but a layer fordetecting blue light is subjected to a certain amount of absorption whenred light or green light passes through, such a light is detected asblue light. Accordingly, even in the event that a blue signal does notactually exist, input of a green or red signal enables a blue signal tobe input, i.e., enables a false signal to be generated, andconsequently, it is difficult to obtain sufficient colorreproducibility.

In order to avoid this situation, signal processing can be made usingcalculations to perform correction as to the entire three primarycolors, which calls for providing additional circuits necessary forcalculations, and consequently, the circuit configuration becomescomplex and great in scale, and also drives up costs. Further, forexample, upon any one of the three primary colors being saturated, itbecomes difficult to recognize the original value of the saturatedlight, and accordingly, the calculations are incorrect, andconsequently, the signal is subjected to processing to obtain color thatis not the original color.

Also, as illustrated in FIG. 1A, most semiconductors have absorptionsensitivity as to infrared light. Accordingly, for example, with a solidstate image capturing apparatus (image sensor) or the like using a Sisemiconductor, it is usually necessary to insert a glass infrared lightcut filter in front of the sensor as an example of subtractive colorfilters.

Accordingly, ways by which to receive infrared light alone, or visiblelight and infrared light as a signal to capture an image includeremoving the infrared light cut filter, or reducing the percentage forcutting infrared light.

However, with such a configuration, infrared light is cast into thephotoelectric conversion device along with the visible light, andaccordingly, the tone of a visible light image differs from the originalone. Accordingly, it is difficult to obtain the respective appropriateimages of a visible image and infrared light alone (or mixture ofinfrared light and visible light) separately at the same time.

Also, apart from the above problems, upon employing an infrared lightcut filter like an ordinary solid state image capturing device, visiblelight is also somewhat cut, resulting in deterioration in sensitivity.Also, employing an infrared light cut filter results in increase incosts.

Also, the arrangements described in Japanese Unexamined PatentApplication Publication No. 10-210486, Japanese Unexamined PatentApplication Publication No. 2002-369049, and Japanese Unexamined PatentApplication Publication No. 06-121325 are wavelength resolution opticalsystems such as a wavelength separating mirror or prism or the like,which causes the input optical system to become great in scale.

Also, the arrangements described in Japanese Unexamined PatentApplication Publication No. 09-166493 and Japanese Unexamined PatentApplication Publication No. 2003-70009 are the insertion/extractionmechanism of an infrared light cut filter, which causes the device tobecome great in scale, and also it is difficult to perform operations ofthe infrared light cut filter automatically.

Also, the arrangement described in Japanese Unexamined PatentApplication Publication No. 09-130678 is a diaphragm optical systemhaving a wavelength resolution function, which causes the device tobecome great in scale. In addition, this arrangement can obtain both ofan infrared image and a visible light image contemporaneously, but onlyan electric signal in which a visible light image and an infrared imageare synthesized can be output from an image sensor, and it is difficultto output a visible light image alone or an infrared image alone.

Also, with the arrangement described in Japanese Unexamined PatentApplication Publication No. 2000-59798, a color difference signal orluminance signal is subjected to correction, but the correction is basedon estimation, resulting in deterioration in correction accuracy.

The present invention has been made in light of the above situations.According to embodiments of the present invention, there is provided anew arrangement for eliminating influence of unnecessary wavelengthregion components such as infrared light, ultraviolet light, and soforth which filter in a principal wavelength region such as a visiblelight region.

According to one embodiment of the present invention, a physicalinformation acquisition method uses a device of which unit componentsinclude detection units for detecting electromagnetic waves and a unitsignal generating unit for generating and outputting the correspondingunit signal based on the amount of electromagnetic waves detected by thedetection unit for detecting a physical quantity distribution in whichthe unit components are disposed on the same substrate in apredetermined sequence, for acquiring physical information for apredetermined application based on the unit signal; wherein a first ofthe detection units detects first wavelength region components followingsecond wavelength region components different from the first wavelengthregion components being separated from the first wavelength regioncomponents beforehand, and also a second of the detection units detectswavelength region components for correction including at least thesecond wavelength region components; and wherein physical informationrelating to the first wavelength region components from which at least apart of influence of the second wavelength region components iseliminated is obtained using the unit signal detected by the firstdetection unit, and the unit signal detected by the second detectionunit.

According to another embodiment of the present invention, a physicalinformation acquisition device, which uses a device of which unitcomponents include a detection unit for detecting electromagnetic waves,and a unit signal generating unit for generating and outputting thecorresponding unit signal based on the amount of electromagnetic wavesdetected by the detection unit for detecting a physical quantitydistribution in which the unit components are disposed on the samesubstrate in a predetermined sequence, for acquiring physicalinformation for a predetermined application based on the unit signal,includes: a first detection unit for detecting first wavelength regioncomponents following second wavelength region components different fromthe first wavelength region components being separated from the firstwavelength region components beforehand; a second detection unit fordetecting wavelength region components for correction including at leastthe second wavelength region components; and a signal processing unitfor obtaining physical information relating to the first wavelengthregion components from which at least a part of influence of the secondwavelength region components is eliminated using the unit signaldetected by the first detection unit, and the unit signal detected bythe second detection unit.

According to another embodiment of the present invention, a physicalinformation acquisition device, which uses a device of which unitcomponents include a detection unit for detecting electromagnetic waves,a unit signal generating unit for generating and outputting thecorresponding unit signal based on the amount of electromagnetic wavesdetected by the detection unit for detecting a physical quantitydistribution in which the unit components are disposed on the samesubstrate in a predetermined sequence, a first detection unit fordetecting first wavelength region components following second wavelengthregion components different from the first wavelength region componentsbeing separated from the first wavelength region components beforehand,and a second detection unit for detecting wavelength region componentsfor correction including at least the second wavelength regioncomponents, for acquiring physical information for a predeterminedapplication based on the unit signal, includes: a signal processing unitfor obtaining physical information relating to the first wavelengthregion components from which at least a part of influence of the secondwavelength region components is eliminated using the unit signaldetected by the first detection unit, and the unit signal detected bythe second detection unit.

According to another embodiment of the present invention, asemiconductor device, which is a device of which unit components includea detection unit for detecting electromagnetic waves and a unit signalgenerating unit for generating and outputting the corresponding unitsignal based on the amount of electromagnetic waves detected by thedetection unit, for detecting a physical quantity distribution in whichthe unit components are disposed on the same substrate in apredetermined sequence, includes on the same substrate: a firstdetection unit for detecting first wavelength region componentsfollowing second wavelength region components different from the firstwavelength region components being separated from the first wavelengthregion components beforehand; and a second detection unit for detectingwavelength region components for correction including at least thesecond wavelength region components.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram describing the arrangements of the sensors describedin U.S. Pat. No. 5,965,875 and Japanese Unexamined Patent ApplicationPublication No. 2004-103964;

FIG. 2 is a diagram illustrating the basic configuration of a layoutexample of color separation filters which enables a visible light colorimage and an infrared light image to be constantly obtainedindependently using correction computing;

FIG. 3 is a diagram illustrating the schematic configuration of an imagecapturing apparatus according to an embodiment of the present invention;

FIGS. 4A and 4B are circuit diagrams of an image capturing apparatus inthe case of applying the layout of the color separation filtersillustrated in FIG. 2 to an interline-transfer-type CCD solid stateimage capturing device;

FIGS. 5A and 5B circuit diagrams of an image capturing apparatus in thecase of applying the layout of the color separation filters illustratedin FIG. 2 to a CMOS solid state image capturing device;

FIGS. 6A and 6B is a diagram for describing one example of the signalacquisition method in the case of employing an image sensor having aconfiguration for separating and obtaining a visible light image and aninfrared light image;

FIGS. 7A and 7B are diagrams for describing a first embodiment of asolid state image capturing device 314;

FIG. 8 is a configuration diagram describing the basic concept of amethod for designing a layered film;

FIG. 9 is a reflectance spectrum diagram describing the basic concept ofa method for designing a layered film;

FIG. 10 is a reflectance spectrum diagram describing the basic conceptof a method for designing a layered film;

FIGS. 11A and 11B are diagrams describing the conditions of a reflectedcenter wavelength λ (diagram illustrating the concept of reflectancespectrums);

FIG. 12 is a reflectance spectrum diagram describing the conditions ofthe reflected center wavelength λ;

FIG. 13 is a reflectance spectrum diagram describing the conditions ofthe reflected center wavelength λ;

FIG. 14 is a configuration diagram describing a first embodiment of aspectral image sensor corresponding to single wavelength divisionutilizing a layered film;

FIG. 15 is a reflectance spectrum diagram describing thicknessdependency according to the first embodiment of a spectral image sensorcorresponding to single wavelength division utilizing a layered film;

FIG. 16 is a diagram (reflectance spectrum diagram; details) describingthe first embodiment of a spectral image sensor corresponding to singlewavelength division utilizing a layered film;

FIG. 17 is a configuration diagram describing the first embodiment of aspectral image sensor corresponding to single wavelength divisionutilizing a layered film;

FIG. 18 is a reflectance spectrum diagram describing the firstembodiment of a spectral image sensor corresponding to single wavelengthdivision utilizing a layered film;

FIG. 19 is a configuration diagram describing a first modification of aspectral image sensor corresponding to single wavelength divisionutilizing a layered film;

FIG. 20 is a reflectance spectrum diagram describing the firstmodification of a spectral image sensor corresponding to singlewavelength division utilizing a layered film;

FIG. 21 is a configuration diagram describing a second modification of aspectral image sensor corresponding to single wavelength divisionutilizing a layered film;

FIG. 22 is a reflectance spectrum diagram describing the secondmodification of a spectral image sensor corresponding to singlewavelength division utilizing a layered film;

FIG. 23 is a configuration diagram describing a third modification of aspectral image sensor corresponding to single wavelength divisionutilizing a layered film;

FIG. 24 is a reflectance spectrum diagram describing the thirdmodification of a spectral image sensor corresponding to singlewavelength division utilizing a layered film;

FIG. 25 is a reflectance spectrum diagram describing thicknessdependency according to the third modification of a spectral imagesensor corresponding to single wavelength division utilizing a layeredfilm;

FIG. 26 is a configuration diagram describing a fourth modification(first thereof) of a spectral image sensor corresponding to singlewavelength division utilizing a layered film;

FIG. 27 is a reflectance spectrum diagram describing the fourthmodification (first thereof) of a spectral image sensor corresponding tosingle wavelength division utilizing a layered film;

FIG. 28 is a configuration diagram describing a fourth modification(second thereof) of a spectral image sensor corresponding to singlewavelength division utilizing a layered film;

FIG. 29 is a reflectance spectrum diagram describing the fourthmodification (second thereof) of a spectral image sensor correspondingto single wavelength division utilizing a layered film;

FIG. 30 is a configuration diagram describing a fifth modification of aspectral image sensor corresponding to single wavelength divisionutilizing a layered film;

FIG. 31 is a reflectance spectrum diagram describing the fifthmodification of a spectral image sensor corresponding to singlewavelength division utilizing a layered film;

FIG. 32 is a diagram illustrating a specific process example formanufacturing a spectral image sensor having a sensor configurationutilizing a layered film;

FIG. 33 is a configuration diagram describing a sixth modification of aspectral image sensor corresponding to single wavelength divisionutilizing a layered film;

FIG. 34 is a configuration diagram describing the sixth modification ofa spectral image sensor corresponding to single wavelength divisionutilizing a layered film;

FIG. 35 is a configuration diagram describing the sixth modification ofa spectral image sensor corresponding to single wavelength divisionutilizing a layered film;

FIG. 36 is a configuration diagram describing the sixth modification ofa spectral image sensor corresponding to single wavelength divisionutilizing a layered film;

FIG. 37 is a diagram describing the sixth modification of a spectralimage sensor corresponding to single wavelength division utilizing alayered film;

FIG. 38 is a reflectance spectrum diagram describing the sixthmodification of a spectral image sensor corresponding to singlewavelength division utilizing a layered film;

FIG. 39 is a reflectance spectrum diagram describing the sixthmodification of a spectral image sensor corresponding to singlewavelength division utilizing a layered film;

FIG. 40 is a diagram for describing a second embodiment of a solid stateimage capturing device;

FIG. 41 is a conceptual diagram describing the basic configuration of aspectral image sensor utilizing a diffraction grating;

FIG. 42 is a diagram enlarging and illustrating one photodiode group ofthe spectral image sensor illustrated in FIG. 41;

FIG. 43 is a diagram describing another embodiment (corresponding toinfrared light) of a spectral image sensor in which a diffractiongrating is disposed at the incident face side of a Si substrate;

FIG. 44 is a chart illustrating the relation between the refractiveindex of Si and the wavelength dispersion of an extinction coefficientwhich are used for a spectral image sensor corresponding to infraredlight;

FIG. 45 is a computing simulation diagram describing a spectral methodin the event that blue light (wavelength of 460 nm) is cast into thelight receiving face of the spectral image sensor having theconfiguration illustrated in FIG. 43;

FIG. 46 is a computing simulation diagram describing a spectral methodin the event that green light (wavelength of 540 nm) is cast into thelight receiving face of the spectral image sensor having theconfiguration illustrated in FIG. 43;

FIG. 47 is a computing simulation diagram describing a spectral methodin the event that red light (wavelength of 640 nm) is cast into thelight receiving face of the spectral image sensor having theconfiguration illustrated in FIG. 43;

FIG. 48 is a computing simulation diagram describing a spectral methodin the event that infrared light (wavelength of 780 nm) is cast into thelight receiving face of the spectral image sensor having theconfiguration illustrated in FIG. 43;

FIG. 49 is a computing simulation diagram describing a spectral methodin the event that infrared light (wavelength of 880 nm) is cast into thelight receiving face of the spectral image sensor having theconfiguration illustrated in FIG. 43;

FIG. 50 is a diagram describing an appropriate example of a detectionposition in dispersion of light between visible light and infrared lightbased on simulation results;

FIG. 51 is a cross-sectional view illustrating one configuration exampleof a sensor configuration corresponding to infrared light correspondingto the detection position in FIG. 50;

FIGS. 52A and 52B are diagrams for describing a third embodiment of asolid state image capturing device;

FIG. 53 is a diagram (part one) describing a problem wherein infraredlight components are mixed in visible light components;

FIG. 54 is a diagram (part two) describing a problem wherein infraredlight components are mixed in visible light components;

FIG. 55 is a diagram (part three) describing a problem wherein infraredlight components are mixed in visible light components;

FIG. 56 is a diagram (part four) describing a problem wherein infraredlight components are mixed in visible light components;

FIG. 57 is a diagram (part five) describing a problem wherein infraredlight components are mixed in visible light components;

FIG. 58 is a diagram (part six) describing a problem wherein infraredlight components are mixed in visible light components;

FIG. 59 is a diagram (part seven) describing a problem wherein infraredlight components are mixed in visible light components;

FIG. 60 is a diagram (part one) describing influence of colorreproducibility caused by infrared light components being mixed invisible light components;

FIG. 61 is a diagram (part two) describing influence of colorreproducibility caused by infrared light components being mixed invisible light components;

FIGS. 62A through 62C are diagrams illustrating the layout of a firstspecific example of a color separating filter for correction computing;

FIG. 63 is a diagram (perspective view) describing a configurationexample of a CCD solid state image capturing device having the layout ofthe color separating filter illustrated in FIGS. 62A through 62C;

FIG. 64 is a diagram (cross-sectional configuration diagram) describinga configuration example of a CCD solid state image capturing devicewhich is configured so as to capture two wavelength components ofinfrared light and visible light separately as images simultaneously;

FIGS. 65A and 65B are diagrams illustrating a properties example of acolor filter to be employed for the first specific example;

FIG. 66 is a diagram (part one) describing a setting method of acoefficient to be employed for correction computing;

FIGS. 67A and 67B are diagrams (part two) describing a setting method ofa coefficient to be employed for correction computing;

FIGS. 68A and 68B are diagrams (part three) describing a setting methodof a coefficient to be employed for correction computing;

FIGS. 69A through 69C are diagrams illustrating the layout of a secondspecific example of a color separating filter for correction computing;

FIG. 70 is a diagram (perspective view) describing a configurationexample of a CCD solid state image capturing device having the layout ofthe color separating filter illustrated in FIGS. 69A through 69C;

FIG. 71 is a diagram describing a correction method of infrared lightcomponents in the second specific example;

FIG. 72 is a diagram (part one) describing the correction technique ofthe third example of the second specific example;

FIG. 73 is a diagram (part two) describing the correction technique ofthe third example of the second specific example;

FIGS. 74A through 74C are diagrams illustrating the layout of a thirdspecific example of a color separating filter for correction computing;

FIG. 75 is a diagram (perspective view) describing a configurationexample of a CCD solid state image capturing device which is configuredso as to have the layout of the color separating filter illustrated inFIGS. 74A through 74C, and capture two wavelength components of infraredlight and visible light separately as images at the same time;

FIGS. 76A and 76B are diagrams (part one) describing a pixel array inlight of deterioration in resolution;

FIG. 77 is a diagram illustrating one example of the transmissionspectral properties of a black filter;

FIGS. 78A and 78B are diagrams (part two) describing a pixel array inlight of deterioration in resolution;

FIGS. 79A through 79C are diagrams (part three) describing a pixel arrayin light of deterioration in resolution;

FIGS. 80A and 80B are diagrams (part four) describing a pixel array inlight of deterioration in resolution;

FIGS. 81A and 81B are diagrams (part five) describing a pixel array inlight of deterioration in resolution;

FIGS. 82A and 82B are diagrams (part six) describing a pixel array inlight of deterioration in resolution;

FIG. 83 is a diagram illustrating the overview of a monochrome cameraemployed for an experiment;

FIG. 84 is a spectral sensitivity properties diagram of the experimentalcamera and color filters;

FIG. 85 is a diagram of the respective transmission spectrums of a blackfilter employed for G color, an infrared light cut filter, andcorrection pixels;

FIG. 86 is a diagram illustrating correspondence of color chip numbers(one cycle; 24 colors) in a Macbeth chart employed as the indices ofcolorimetry;

FIGS. 87A through 87C are diagrams illustrating the image based onunfiltered image data obtained by capturing a Macbeth chart using theexperimental camera and a green filter G;

FIG. 88 is a diagram representing a signal level (actual value) for eachcolor chip number of the Macbeth chart which is the image capturingresult illustrated in FIGS. 87A through 87C;

FIGS. 89A and 89B are diagrams illustrating the image based onunfiltered image data obtained by capturing a Macbeth chart using theexperimental camera and a black filter BK serving as a correction pixel;

FIGS. 90A and 90B are diagrams illustrating a black correction imageobtained by multiplying a black filter image by a predeterminedcoefficient;

FIGS. 91A through 91C are diagrams representing an image illustratingone example of correction advantages as to a G color image employing ablack correction image;

FIG. 92 is a diagram representing a signal level (actual value) for eachcolor chip number of the Macbeth chart which illustrates one example ofcorrection advantages as to a G color image employing a black correctionimage;

FIGS. 93A and 93B are diagrams representing an image illustrating theadvantages of a high-precision correction method as to a G color imageemploying a black correction image;

FIG. 94 is a diagram representing a signal level (actual value) for eachcolor chip number of the Macbeth chart of a high-precision correctionmethod as to a G color image employing a black correction image;

FIG. 95 is a diagram illustrating environmental conditions at the timeof an experiment in the case of applying white correction pixels;

FIG. 96 is a diagram illustrating the transmission properties of anordinary IR cut filter and a pseudo MLT filter;

FIG. 97 is a spectral sensitivity properties diagram of color filters inthe case of applying the experimental camera and the pseudo MLT filter;

FIG. 98 is a flowchart illustrating the overall procedures;

FIG. 99 is a diagram illustrating the results of capturing 24 colors ofa Macbeth chart under an environment of a halogen light source (colortemperature of 3000 K), and obtaining the color difference before andafter correction by computing;

FIG. 100 is a chart summarizing the measurement result of colordifference as to each type of light source;

FIG. 101 is a chart summarizing the estimated values and actual valuesof noise regarding the halogen light source (color temperature of 3000K); and

FIG. 102 is a chart summarizing the estimated values and actual valuesof noise regarding fluorescent light.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will be described below in detailwith reference to the drawings.

<Basic Concept>

FIG. 2 is a diagram illustrating the basic configuration of a layoutexample of color separation filters which constantly enable a visiblelight color image and an infrared light image to be obtainedindependently using correction computing. Here, four types of colorfilters having distinct filter properties are disposed regularly (atetragonal lattice in the present embodiment), which are made up ofcolor filters C1, C2, and C3 for visible light color images (any ofthese transmits the first wavelength region components) serving asfilters for three wavelength regions (color components), and a colorfilter C4 for infrared light serving as second wavelength regioncomponents different from the components of the color filters C1, C2,and C3. The respective components can be detected independently by thecorresponding detection unit through the color filters C1, C2, C3, andC4. The detection unit in which the color filters C1, C2, and C3 aredisposed is a first detection unit, and the detection unit in which thecolor filter C4 is disposed is a second detection unit. Also, thedetection unit (detection elements) in which the color filters C1, C2,and C3 are disposed is a detection unit for detecting a first wavelengthregion by further subjecting the first wavelength region to wavelengthseparation.

Let us say that the color filters C1, C2, and C3 are primary colorfilters wherein the transmittance of color components within a visiblelight band is generally one, and the others are generally zero. Forexample, primary color filters may be employed wherein a blue componentB (e.g., transmittance is generally one at wavelength λ=400 through 500nm, and generally zero at the others), a green component G (e.g.,transmittance is generally one at wavelength λ=500 through 600 nm, andgenerally zero at the others), and a red component R (e.g.,transmittance is generally one at wavelength λ=600 through 700 nm, andgenerally zero at the others), which are three primary colors of visiblelight VL (wavelength λ=380 through 780 nm), are taken as the center.

Alternatively, let us say that the color filters C1, C2, and C3 arecomplementary-color filters having transmittance of generally zero forcolor components within a visible light band, and generally one for theothers. For example, complementary-color filters having transmittance ofgenerally zero as to the three primary color components of visible lightmay be employed, such as yellow Ye (e.g., transmittance is generallyzero at wavelength λ=400 through 500 nm, and generally one at theothers), magenta Mg (e.g., transmittance is generally zero at wavelengthλ=500 through 600 nm, and generally one at the others), cyan Cy (e.g.,transmittance is generally zero at wavelength λ=600 through 700 nm, andgenerally one at the others), and so forth.

Complementary-color filters have higher sensitivity than primary-colorfilters, whereby the sensitivity of an image capturing device can beimproved by employing complementary-color filters wherein thetransmitted light of a visible region is made up of complementary colorsas to each of the three primary colors. Inversely, employingprimary-color filters provides advantages wherein primary color signalscan be obtained without performing difference processing, and the signalprocessing of a visible light color image can be readily performed.

Note that transmittance of “generally one” shows an ideal state, and itis desirable that the transmittance in the wavelength region isextremely greater than that in the other wavelength regions. A partthereof may include transmittance other than “1”. Similarly,transmittance of “generally zero” also shows an ideal state, and it isdesirable that the transmittance in the wavelength region is extremelysmaller than that in the other wavelength regions. A part thereof mayinclude transmittance other than “zero”.

Also, any of primary-color filters and complementary-color filters canbe employed as long as filters pass through the wavelength regioncomponents of a predetermined color within a visible light region(primary color or complementary color), and whether or not the filterspass through an infrared light region serving as a second wavelengthregion, i.e., transmittance as to infrared light IR can be ignored.

For example, with the respective color filters currently commonlyemployed, for example, transmittance is high as to each of R, G, and B,and transmittance is low as to the other colors (e.g., G or B in thecase of R) within a visible light band, but regarding the transmittanceother than a visible light band is irregular, and is usually higher thanthe transmittance of the other colors (e.g., G or B in the case of R),e.g., each filter has sensitivity as to an infrared region, and there istransmittance of light in an infrared region. However, even if thetransmittance other than a visible light band is high, the presentembodiment does not receive influence thereof.

On the other hand, the color filter C4 is preferably a filter for apredetermined wavelength region including at least the components of thesecond wavelength region (infrared light in the present example), may bea filter (visible light cut filter) serving as a first technique for notpassing through principal components passing through the color filtersC1, C2, and C3 (i.e., visible light components) but passing through onlythe second wavelength region (infrared light in the present example), ormay be a filter (whole-region passage filter) serving as a secondtechnique for passing through the entire regions components from thefirst wavelength region (visible light in the present example) to thesecond wavelength region (infrared light in the present example). In thecase of the second technique, in that the entire wavelength componentsfrom visible light to infrared light (particularly near-infrared light)are passed through, as a matter of fact, a configuration in which nocolor filter is provided can be employed as the color filter C4.

For example, in the event that a visible light cut filter is notinserted at the light receiving face side of a detection region whichreceives infrared light, visible light components are filtered in thedetection region side of infrared light, and the visible light image ofthe filtered components and the original infrared light image areobtained in a mixed manner. In order to eliminate this mixed visiblelight image, and obtain an infrared light image receiving almost noinfluence from visible light, for example, it is necessary to reduce theintensity of blue, red, and green to be detected at three color pixelsR, G, and B which receive visible light components.

Conversely, for example, providing a green filter passing through redlight and green light as a visible light cut filter enables the mixedcomponents of infrared light IR and green visible light LG to beobtained from the detection unit for infrared light, but taking thedifference as to the green components obtained from the color pixel Greceiving green light components enables an image of only the infraredlight IR receiving almost no influence from visible light VL (here,green light G) to be obtained. It is necessary to provide a green filterat the light receiving face side of a detection unit for outside light,but this facilitates processing as compared with the case of subtractingthe intensity of blue, red, and green to be detected at the three pixelsR, G, and B without providing a green filter.

Also, upon providing a black filter such as passing through infraredlight, and absorbing visible light alone as a visible light cut filter,the components of infrared light alone can be obtained from thedetection unit for infrared light by this black filter absorbing visiblelight, and even if difference processing is not performed, an infraredlight image of infrared light alone receiving almost no influence ofvisible light can be obtained.

Note that the pixel detection unit where the color filters C1, C2, andC3 are disposed (e.g., image capturing device such as a photodiode)preferably has sensitivity as to visible light, and has no necessity tohave sensitivity as to near-infrared light. On the other hand, it isnecessary for the detection unit configured of a pixel photodiode wherethe color filter C4 is disposed to have sensitivity as to at leastnear-infrared light in the case of the present example. Also, in theevent that the color filter C4 is a visible light cut filter passingthrough near-infrared light alone, it is unnecessary to have sensitivityas to visible light, but in the event that the color filter C4 is awhole-region passage filter, it is also necessary to have sensitivity asto visible light.

Also, color pixels where the color filter C4 is disposed are employednot only as physical information (infrared light image in the presentexample) for reproducibility relating to the second wavelength regioncomponents to be obtained based on the color pixels where the colorfilter C4 is disposed but also as correction pixels as to a color signalfor visible light color image reproducibility to be obtained based onthe color pixels where the color filters C1, C2, and C3 are disposed.Consequently, the color filter C4 serves as a correction color filter asto the color filters C1, C2, and C3.

That is to say, as for reproducibility of a visible light color image,first, the signal components SC1, SC2, and SC3 of the first wavelengthregion are separated from the second wavelength region (infrared)components which are different from the first wavelength regioncomponents based on the color pixels where the color filters C1, C2, andC3 are disposed as a matter of fact, and detection is performed in eachindependent detection region. Also, the signal component SC4 of apredetermined wavelength region (infrared alone or entire region)including at least the second wavelength region (infrared) components isdetected in further another detection region.

The respective signal components SC1, SC2, and SC3 are corrected usingthe signal component SC4, thereby obtaining respective correction colorsignals SC1*, SC2*, and SC3* for reproducing an image (here, visiblelight color image) relating to the first wavelength region components(visible light components) from which influence of the second wavelengthregion (infrared) components is eliminated.

As for this correction computing, signal components obtained bymultiplying the signal component SC4 including at least the secondwavelength components by predetermined coefficients αC1, αC2, and αC3 issubtracted from the signal components SC1, SC2, and SC3 of the firstwavelength region.

Note that an image relating to the second wavelength region components(here, infrared light image relating to infrared light) can be obtainedfrom the signal component SC4. At this time, in the event that the colorfilter C4 is a visible light cut filter which does not pass throughprincipal components (i.e., visible light components) passing throughthe color filters C1, C2, and C3 but the second wavelength region(infrared light in the present example) alone, the signal component SC4itself represents an infrared light image. On the other hand, in theevent that the color filter C4 is a whole-region passage filter passingthrough the entire region components from the first wavelength region(visible light in the present example) to the second wavelength region(infrared light in the present example), visible light image componentsto be obtained by the signal components SC1, SC2, and SC3 should besubtracted from the signal component SC4. Note that as for an infraredlight image, an image wherein infrared light and visible light are mixedmay be obtained.

Thus, upon signal output to be obtained from the four types ofwavelength regions (here, respective pixels where the four types ofcolor filters are disposed) being subject to matrix computing, each of avisible light color image and a near-infrared light image can beobtained independently. That is to say, the four types of color filtershaving different filter properties are disposed at the respective pixelsof an image capturing device such as a photodiode, and the output ofeach pixel where the four types of color filters are disposed issubjected to matrix computing, whereby three primary output for forminga visible color image receiving almost no influence of near-infraredlight, and output for forming a near-infrared light image receivingalmost no influence of visible light can be obtained independently andalso simultaneously.

Particularly, as for a visible light color image, deterioration in colorreproducibility due to filtering of infrared light is corrected withcomputing processing, whereby image capturing having high sensitivity ata dark place and also excellent color reproducibility can be performed.A phenomenon wherein a red signal component close to infrared lightbecomes great, and a phenomenon wherein luminance at a red portion of animage increases can be absorbed, and also improvement of colorreproducibility and sensitivity rise at the time of low illumination canbe balanced without employing a special image capturing device andmechanism.

Also, it becomes unnecessary to insert an expensive glass optical memberhaving certain thickness and weight serving as one example of asubtractive filter in front of a sensor on the optical path of animage-forming optical system. The optical system can be reduced inweight and sized, and costs can be greatly reduced by eliminating thenecessity of an expensive infrared light cut filter. It is needless tosay that the insertion/extraction mechanism of an infrared light cutfilter is unnecessary, which prevents the device from becoming great inscale.

High sensitivity can be also realized by eliminating the necessity of aninfrared light cut filter. Performing color image capturing without aninfrared light cut filter enables current signal processing circuits tobe combined, and also enables the light of a near-infrared region to beused effectively, whereby high sensitivity can be realized, and at thattime, excellent color reproducibility can be obtained even when lowillumination.

Deterioration in color reproducibility of a visible light color imagedue to infrared light components to be filtered in visible lightcomponents can be readily corrected with computing processing. Also, asfor the correction computing, correction is made not by simpleestimation such as the arrangement described in Japanese UnexaminedPatent Application Publication No. 2003-70009 but by actually measuringinfrared light components, and using the information thereof, wherebycorrection can be made for the appropriate amount depending on theintensity of infrared light under an actual image capturing environment,and consequently, correction accuracy is extremely in a good condition.Also, it is unnecessary for a user to adjust the amount of correctioncorresponding to an image capturing environment, thereby obtaining easeof use.

<Image Capturing Apparatus>

FIG. 3 is a diagram illustrating the schematic configuration of an imagecapturing apparatus serving as one example of a physical informationacquisition device according to an embodiment of the present invention.This image capturing apparatus 300 is an image capturing apparatus forobtaining a visible light color image and a near-infrared light imageindependently.

Specifically, the image capturing apparatus 300 includes a taking lens302 for guiding light L which holds the image of a subject Z to theimage capturing unit side to form an image, an optical lowpass filter304, an image capturing unit 310 having a color filter group 312 and asolid state image capturing device (image sensor) 314, a driving unit320 for driving the solid state image capturing device 314, respectiveimage capturing signals SIR (infrared light components) output from thesolid state image capturing device 314, and an image capturing signalprocessing unit 330 for processing SV (visible light components).

The optical lowpass filter 304 is for shielding high-frequencycomponents exceeding Nyquist frequency to prevent loop distortion. Acommon image capturing apparatus uses the optical lowpass filter 304 andan infrared light cut filter together, but the present configurationincludes no infrared light cut filter. Also, in the event of employing aconfiguration wherein a visible light color image and a near-infraredlight image are independently obtained, an arrangement is sometimesemployed wherein an optical member for wavelength separation (referredto as wavelength separating optical system) is provided for separatingthe light L1 to be cast through the taking lens 302 into an infraredlight IR serving as one example of invisible light and a visible lightVL, but the present configuration includes no wavelength separatingoptical system for performing wavelength separation in such an incidentsystem.

The solid state image capturing device 314 is an image capturing devicemade up of a photoelectric conversion pixel group formed in atwo-dimensional matrix form. Note that the specific configuration of thesolid state image capturing device 314 to be employed for the presentembodiment will be described later.

Electric charge corresponding to the infrared light IR for holding theimage of the subject Z, and electric charge corresponding to the visiblelight VL occur on the image capturing face of the solid state imagecapturing device 314. Operation such as accumulation operation ofelectric charge, readout operation of electric charge, and so forth iscontrolled by a sensor driving pulse signal to be output to the drivingunit 320 from an unshown system control circuit.

The electric charge signals read out from the solid state imagecapturing device 314, i.e., the infrared light image capturing signalSIR holding an infrared light image, and the visible light imagecapturing signal SVL holding a visible image are transmitted to theimage capturing signal processing unit 330, and are subjected topredetermined signal processing.

For example, the image capturing signal processing unit 330 includes apre-processing unit 332 for subjecting the sensor output signals(visible light image capturing signal SVL and infrared light imagecapturing signal SIR) output from the solid state image capturing device314 to pre-processing such as black-level adjustment, gain adjustment,gamma correction, or the like, an AD conversion unit 334 for convertingthe analog signal output from the pre-processing unit 332 into a digitalsignal, a correction processing unit 336 for correcting shading whichoccurs at the taking lens 302, the pixel defect of the solid state imagecapturing device 314, and so forth, and an image signal processing unit340.

The image signal processing unit 340 serving as the feature portion ofthe present embodiment includes an infrared light correction processingunit 342 for generating a correction visible light image capturingsignal SVL* by subjecting the visible light image capturing signal SVLto correction using the infrared light image capturing signal SIR. Also,the image signal processing unit 340 includes a luminance signalprocessing unit 344 for generating a luminance signal based on thecorrection visible light image capturing signal SVL* output from theinfrared light correction processing unit 342, a color signal processingunit 346 for generating a color signal (primary-color signal orcolor-difference signal) based on the correction visible light imagecapturing signal SVL* output from the infrared light correctionprocessing unit 342, and an infrared signal processing unit 348 forgenerating an infrared light signal representing an infrared light imagebased on the infrared light image capturing signal SIR.

The image capturing signal output from the solid state image capturingdevice 314 is amplified into a predetermined level by the pre-processingunit 332 of the image capturing signal processing unit 330, and isconverted into a digital signal from the analog signal by the ADconversion unit 334. Also, the digital image signal of visible lightcomponents of which infrared light components are suppressed by theinfrared light correction processing unit 342, and is further separatedinto color separation signals of R, G, and B as necessary (particularly,in the event of employing complementary-color filters as the colorfilters C1, C2, and C3) by the luminance signal processing unit 344 andthe color signal processing unit 346, following which are converted intoa luminance signal, a color signal, or a picture signal obtained bysynthesizing the luminance signal and color signal, and are output.Also, the infrared signal processing unit 348 subjects the infraredlight image capturing signal SIR to correction using the visible lightimage capturing signal SVL as necessary (in the event that a blackfilter is not used as the color filter C4).

Note that with the infrared light correction processing unit 342, theplacement position thereof is not restricted to such a configuration aslong as the infrared light correction processing unit 342 can subjectthe visible light image capturing signal SVL to correction using theinfrared light image capturing signal SIR. For example, an arrangementmay be made wherein the infrared light correction processing unit 342 isprovided between the AD conversion unit 334 and the correctionprocessing unit 336 for shading correction or pixel defect correctionare provided, and correction for suppressing influence of infrared lightis performed prior to shading correction and pixel defect correction.

Alternatively, an arrangement may be made wherein the infrared lightcorrection processing unit 342 is provided between the pre-processingunit 332 and the AD conversion unit 334, infrared light suppressingprocessing is performed following pre-processing such as black-leveladjustment, gain adjustment, gamma correction, or the like, or whereinthe infrared light correction processing unit 342 is provided betweenthe solid state image capturing device 314 and the pre-processing unit332, infrared light suppressing processing is performed prior topre-processing such as black-level adjustment, gain adjustment, gammacorrection, or the like.

With such a configuration, with the image capturing apparatus 300, thetaking lens 302 takes in an optical image representing the subject Zincluding the infrared light IR, the image capturing unit 310 takes inan infrared light image (near-infrared light optical image) and avisible light image (visible light optical image) without separation,the image capturing signal processing unit 330 converts these infraredlight image and visible light image into picture signals respectively,following which subjects the picture signals to predetermined signalprocessing (e.g., color signal separation to R, G, and B components, orthe like), and outputs these as a color image signal or infrared lightimage signal, or a mixed image signal in which both are synthesized.

For example, the taking lens 302 is a lens made up of an opticalmaterial such as quartz, sapphire, or the like which can transmit lightof which wavelength is around 380 nm through around 2200 nm, which takesin an optical image including the infrared light IR, and whilecondensing this forms an image on the solid state image capturing device314.

Also, the image capturing apparatus 300 according to the presentembodiment has features in that a detection unit (image sensor), whichis optimized so as to detect the wavelength components of the originaldetection target, is provided in the image capturing unit 310.Particularly, with the present embodiment, the solid state imagecapturing device 314, which is optimized so as to detect the visiblelight VL, is provided to detect short wavelength sides within thevisible light VL and the infrared light IR.

Now, the term “image sensor which is optimized” means to have aconfiguration wherein a region corresponding to wavelength separation isprovided, in which other than the wavelength components of the originaldetection target, are not included in the image capturing signal of thewavelength components of the original detection target, as much aspossible.

The present embodiment has features in that the image sensor side has aconfiguration corresponding to wavelength separation even if awavelength separating optical system has no wavelength separation on theoptical path, whereby the optical system can be reduced in size.

The configuration of such an image capturing apparatus differs from aconfiguration wherein as described in Japanese Unexamined PatentApplication Publication No. 10-210486 and Japanese Unexamined PatentApplication Publication No. 06-121325, a visible light image and aninfrared light image are individually obtained by the respectivewavelength components separated by the wavelength separating opticalsystem being cast into the respective individual sensors having the sameconfiguration.

Also, the configuration of the above image capturing apparatus differsfrom an arrangement wherein as described in Japanese Unexamined PatentApplication Publication No. 10-210486, the individual images of R, G,and B are obtained regarding the visible light VL by further separatingthe visible light components which transmitted a cold mirror into a redcomponent, a green component, and a blue component at three dichroicmirrors, and casting each of these into an individual sensor. With themethod in Japanese Unexamined Patent Application Publication No.10-210486, it is necessary to provide three sensors regarding thevisible light VL, which improves sensitivity, but also poses a problemsuch as increase in costs. The configuration according to the presentembodiment does not have such a problem.

Also, the configuration of the above image capturing apparatus differsfrom a configuration wherein as described in Japanese Unexamined PatentApplication Publication No. 2002-369049, a visible light image and aninfrared light image are individually obtained by performing wavelengthseparation with a two-step process on the optical path, and casting eachseparated component into an individual sensor having the sameconfiguration. With the method in Japanese Unexamined Patent ApplicationPublication No. 2002-369049, wavelength separation is performed with atwo-step process on the optical path, which creates a drawback whereinthe optical system becomes great in scale. In addition, problems such assensitivity, blurring, and so forth are included. The configurationaccording to the present embodiment does not have such a problem.

For example, with the configuration of the present embodiment, itbecomes unnecessary to insert an infrared light cut filter serving asone example of a subtractive filter in front of the solid state imagecapturing device 314 at the time of image capturing of the visible lightVL using the image capturing unit 310. Costs can be greatly reduced byeliminating the necessity of such an expensive infrared light cutfilter. Also, an optical system can be reduced in weight and also insize by eliminating the necessity of an infrared light cut filter havingthickness and weight. It is needless to say that theinsertion/extraction mechanism of an infrared light cut filter isunnecessary, which prevents the device from becoming great in scale. Theconfiguration of the present embodiment is cost-wise advantageous ascompared with the case of employing an existing glass infrared light cutfilter, and also can provide an image capturing apparatus such as adigital camera and so forth which has become compact and excels inportability and so forth.

Also, with a configuration wherein an infrared light cut filter isinserted in front of the solid state image capturing device 314,interface between the air and the glass occurs on the way to the opticalpath by inserting a glass substrate in front of an image capturingdevice such as CCD, CMOS, or the like. Accordingly, even the light ofthe visible light VL which is desired to be transmitted is alsoreflected at the interface thereof, which poses a problem of leading todeterioration in sensitivity. Further, upon the number of such aninterface increasing, an angle to be refracted in oblique incidence(within glass) differs depending on a wavelength, leading to defocus dueto change in the optical path. Conversely, employing no infrared lightcut filter on the optical path of the front side of the solid stateimage capturing device 314, enables an advantage wherein such defocus iseliminated to be obtained.

Note that in order to further improve wavelength separation performance,weak infrared light cut filters may be inserted entirely, though aproblem such as increase in size of the optical system, and so forthoccurs. For example, up to a level having almost no problem as to thevisible light VL may be cut by inserting an infrared light cut filter of50% or less.

Either way, image capturing of the visible light VL alone and imagecapturing of the infrared light IR alone, or image capturing of thevisible light VL alone and image capturing in which the infrared lightIR and the visible light VL are mixed can be performedcontemporaneously.

According to the present embodiment, influence of the infrared light IRis not received at the time of image capturing of a monochrome image ora color image during the day, and also image capturing using theinfrared light IR can be performed during the night or the like. Theother images can be output contemporaneously as necessary. Even in sucha case, the image of the infrared light IR alone which receives noinfluence from the visible light VL can be obtained.

For example, the monochrome image of the visible light VL alone whichalmost completely receives no influence from the infrared light IR canbe obtained. The present embodiment differs from the arrangementdescribed in Japanese Unexamined Patent Application Publication No.2002-142228, it is unnecessary to perform computing processing among thecomponents of the infrared light IR at the time of obtaining themonochrome image of the visible light VL which almost completelyreceives no influence from the infrared light IR.

Further, as for one example of an optical member for separating thevisible light VL into predetermined wavelength region components, colorfilters having predetermined wavelength transmission properties in avisible light region are provided corresponding to pixels (unit pixelmatrix), whereby the image of the particular wavelength region alonewithin a visible light region which almost completely receives noinfluence from the infrared light IR can be obtained.

Also, color filters each having different wavelength transmissionproperties in a visible light region are arrayed regularly on multiplephotodiodes making up a unit pixel matrix by integrally matching theposition thereof to the photodiode corresponding to each wavelength (bycolor), whereby the visible light region can be separated by wavelength(by color). The color image (visible light color image) of the visiblelight VL alone which almost completely receive no influence from theinfrared light IR can be obtained by performing synthetic processingbased on the respective pixel signals to be obtained from these pixelsby color.

In the event of obtaining the color image of the visible light VL whichalmost completely receives no influence from the infrared light IR,correction computing for subtracting signal components obtained bymultiplying the signal components SIR including at least infrared lightregion components by a predetermined coefficient α from the visiblelight region signal components SV, which is different from a simplematrix computing such as the arrangement described in JapaneseUnexamined Patent Application Publication No. 2002-142228, whereby theinfrared light components included in a visible region pixel signal canbe suppressed with high precision.

Also, infrared light components are actually measured, and visible lightcomponents are subjected to correction using that information, which isdifferent from simple estimation correction such as the arrangementdescribed in Japanese Unexamined Patent Application Publication No.2003-70009, whereby correction can be performed depending on the actualsituation with high precision.

Thus, the monochrome image or color image of the visible light VL, andthe “image relating to the infrared light IR” can be independentlyobtained all the time. The term “the image relating to the infraredlight IR” means the image of the infrared light IR alone which almostcompletely receives no influence from the visible light VL, or the imagewherein the infrared light IR and the visible light VL are mixed.

An arrangement may be made wherein image capturing of the visible lightVL alone (monochrome image capturing or color image capturing) whichalmost completely receives no influence from the infrared light IR, andimage capturing wherein the infrared light IR and the visible light VLare mixed can be performed contemporaneously. Also, an arrangement maybe made wherein image capturing of the infrared light IR alone whichalmost completely receives no influence of the visible light VL can beperformed by performing synthetic processing (specifically, differenceprocessing) between the components of the visible light VL alone(monochrome image components or color image components) and thecomponents wherein the infrared light IR and the visible light VL aremixed.

Note that the above term “almost completely receives no influence” maybe “receives somewhat influence” conclusively in light of human visiongenerally as long as the degree of influence is that it is difficult tosense apparent difference by human vision. That is to say, as for theinfrared light IR side, it is desirable to obtain an infrared image (oneexample of physical information) of which influence of a passagewavelength region (visible light VL) can be ignored from the perspectiveof human vision, and as for the visible light VL side, it is desirableto obtain an ordinary image (one example of physical information) ofwhich influence of reflective wavelength region components (infraredlight IR) can be ignored from the perspective of human vision.

Note that in the event of employing no black filter as the color filterC4, the correction pixels where the color filter C4 is disposed havesensitivity as to a wide wavelength area from visible light to infraredlight, and accordingly, the pixel signals thereof are readily saturatedas compared with the other pixels for visible light image capturingwhere the color filters C1, C2, and C3 are disposed.

In order to avoid this problem, it is desirable for the driving unit 320to control the detection time of the second detection unit where thecolor filter C4 is disposed. For example, with image capturing at abright place, it is desirable to read out a pixel signal from thecorrection pixel detection unit in shorter a cycle than an ordinarycycle using an electronic shutter function or the like, and transmitthis to the pre-processing unit 332. In this case, an advantage can beobtained as to saturation by transmitting the signal in higher a ratethan 60 frame/sec.

Alternatively, it is desirable to read out electric charge from thecorrection pixel detection unit in shorter time (accumulation time) than0.01667 sec. In this case, accumulation of electric charge may be readout effectively in a short period of time by discharging an electriccharge signal to the substrate side using overflow. Further preferably,an advantage can be obtained as to saturation by transmitting the signalin higher a rate than 240 frame/sec. Alternatively, it is desirable tosimply read out electric charge from the correction pixel detection unitin shorter time (accumulation time) than 4.16 ms. Either way, it isdesirable to prevent the pixel signal to be output from the correctionpixel detection unit from saturation. Note that thus, pixels of whichelectric charge is read out in a short period of time (accumulationtime) so as to prevent saturation may be correction pixels alone, or maybe all of the pixels.

Further, a weak signal may be converted into a strong signal to improvean S/N ratio by integrating the signal read out in a short period oftime twice or more. For example, according to such an arrangement, evenif image capturing is performed at a dark place or bright place,appropriate sensitivity and a high S/N ratio can be obtained, leading toexpand a dynamic range.

<Image Capturing Apparatus; Corresponding to CCD>

FIGS. 4A and 4B circuit diagrams of an image capturing apparatus in thecase of applying the layout of the color separation filters illustratedin FIG. 2 to an interline-transfer-type CCD solid state image capturingdevice (IT_CCD image sensor).

Now, FIGS. 4A and 4B illustrate a configuration wherein the infraredlight IR is detected while separating the inside of a visible light bandinto the respective color components of R, G, and B, which is aconfiguration wherein blue light B, green light G, and red light Rwithin the visible light VL, and the infrared light IR are each detectedindependently, and substantially, which is a configuration whereinwithin one unit pixel matrix 12 pixels (photoelectric conversiondevices) 12B, 12G, and 12R are formed by wavelength, and also a pixel12IR having no wavelength separation configuration is included, and thepixel 12IR is used as a correction pixel as to the other pixels.

For example, as illustrated in FIG. 4A, with a CCD solid state imagecapturing device 101, in addition to the unit pixel matrix 12, aplurality of vertical transfer CCDs 122 are arrayed in the verticaltransfer direction. The electric charge transfer direction of thevertical transfer CCDs 122, i.e., the readout direction of a pixelsignal is the vertical direction (X direction in the drawing).

Further, MOS transistors making up readout gates 124 (124B, 124G, 124R,and 124IR by wavelength) stand between the vertical transfer CCD 122 andeach unit pixel matrix 12, and also an unshown channel stop is providedat the interface portion of each unit cell (unit component).

Note that as can be understood from FIGS. 4A and 4B, the one unit pixelmatrix 12 is configured so as to detect the blue light B, green light G,red light R, and infrared light IR independently, and substantially,which is a configuration wherein the pixels 12B, 12G, 12R, and 12IR areformed by wavelength (color) within the one unit pixel matrix 12. Animage capturing area 110 is configured of the plurality of verticaltransfer CCDs 122, which are provided for each vertical row of sensorunits 112 made up of the unit pixel matrices 12, for subjecting thesignal electric charge read out by the readout gates 124 from eachsensor unit to vertical transfer, and the sensor units 112.

Now, as for the array of the color filters 14, for example, let us saythat the sequence in the vertical direction (X direction) of thevertical transfer CCD 122 at the light receiving face side of a siliconsubstrate 1_ω is blue, green, red, IR (correction pixel), blue, green,red, IR (correction pixel) and so on, and similarly, the sequence in thesame line direction (Y direction) of the plurality of vertical transferCCDs 122 is blue, green, red, IR (correction pixel), blue, green, red,IR (correction pixel), and so on. Also, it is effective to employ apixel array in light of deterioration in resolution by providingcorrection pixels (detailed description will be made later).

The signal electric charge accumulated in the unit pixel matrices 12(each of the pixels 12B, 12G, 12R, and 12IR) of the sensor units 112 isread out by the vertical transfer CCD 122 of the same vertical row by adrive pulse φROG corresponding to a readout pulse ROG being applied tothe readout gates 124. The vertical transfer CCD 122 is subjected totransfer driving by a drive pulse φVx based on vertical transfer clockVx such as three phases through eight phases, for example, and transfersthe readout signal electric charge in the vertical direction for eachportion equivalent to one scan line (one line) during a part of ahorizontal blanking period in order. This vertical transfer for each oneline is particularly referred to as “line shift”.

Also, with the CCD solid state image capturing device 101, a horizontaltransfer CCD 126 (H register unit, horizontal transfer unit) is providedfor the worth of one line, which is adjacent to each of the transferdestination side end portion of the plurality of vertical transfer CCDs122, i.e., the vertical transfer CCD 122 of the last line, and extendsin a predetermined (e.g., horizontal) direction. This horizontaltransfer CCD 126 is subjected to transfer driving by drive pulses φH1and φH2 based on two-phase horizontal transfer clocks H1 and H2 forexample, and transfers the signal electric charge for the worth of oneline transferred from the plurality of vertical transfer CCDs 122 to thehorizontal direction in order during a horizontal scan period followinga horizontal blanking period. Accordingly, a plurality (two) ofhorizontal transfer electrodes corresponding to two-phase driving areprovided.

The transfer destination end portion of the horizontal transfer CCD 126is provided with an output amplifier 128 including an electric-chargevoltage conversion unit having a floating diffusion amplifier (FDA)configuration, for example. The output amplifier 128 is one example of aphysical information acquisition unit, which converts the signalelectric charge subjected to horizontal transfer by the horizontaltransfer CCD 126 into a voltage signal sequentially at theelectric-charge voltage conversion unit, amplifies this to apredetermined level, and outputs this. With this voltage signal, a pixelsignal is exhausted as CCD output (Vout) depending on the incidentamount of light from a subject. Thus, the interline-transfer-type CCDsolid state image capturing device 101 is configured.

The pixel signal exhausted from the output amplifier 128 as CCD output(Vout) is input to the image capturing signal processing unit 330, asillustrated in FIG. 4B. An image switchover control signal from an imageswitchover control unit 360 serving as one example of a signalswitchover control unit is configured so as to be input to the imagecapturing signal processing unit 330.

The image switchover control unit 360 instructs switchover regardingwhether the output of the image capturing signal processing unit 330 ischanged to either of the monochrome image or color image of the visiblelight VL which almost completely receive no influence from the infraredlight IR, and the image of the infrared light IR which almost completelyreceive no influence from the visible light VL, or both of these, or themixed image of the visible light VL and the infrared light IR, i.e., thepseudo monochrome image or pseudo color image to which the luminance ofthe infrared light IR is added. That is to say, the image switchovercontrol unit 360 controls simultaneous image capturing output betweenthe image of the visible light VL and the image according to theinfrared light IR, and switchover image capturing output.

This command may be provided with external input which operates theimage capturing apparatus, or the image switchover control unit 360 mayissue the command of switchover by automatic processing depending on thevisible light luminance including no infrared light IR of the imagecapturing signal processing unit 330.

Now, the image capturing signal processing unit 330 performs, forexample, synchronization processing for synchronizing the imagecapturing data R, G, B, and IR of each pixel, pinstriped noisecorrection processing for correcting pinstriped noise components causedby smear phenomenon or blooming phenomenon, white balance controlprocessing for controlling white balance (WB) adjustment, gammacorrection processing for adjusting gradation degree, dynamic rangeenlargement processing for enlarging a dynamic range using pixelinformation of two screens having different electric charge accumulationtime, YC signal generation processing for generating luminance data (Y)and color data (C), or the like. Thus, a visible light band image(so-called ordinary image) based on the primary-color image capturingdata (each pixel data of R, G, B, and IR) of red (R), green (G), andblue (B) can be obtained.

Also, the image capturing signal processing unit 330 generates an imagerelating to the infrared light IR using the pixel data of the infraredlight IR. For example, with the pixel 12IR which serves as a correctionpixel as to the pixels 12R, 12G, and 12B for visible light imageacquisition, in the event that the color filter 14C is not inserted suchthat not only the infrared light IR but also the visible light VLcontribute to a signal simultaneously, an image having high sensitivitycan be obtained by using the pixel data from the pixel 12IR. Also, animage of the infrared light IR alone can be obtained by taking thedifference as to each color component to be obtained from the pixels12R, 12G, and 12B.

In the event of inserting the green filter 14G as to the pixel 12IR asthe color filter 14C, an image in which the infrared light IR and thegreen visible light LG are mixed is obtained from the pixel 12IR, but animage of the infrared light IR alone is obtained by taking thedifference as to the green components to be obtained from the pixel 12G.Alternatively, in the event of providing a black filter 14BK as to thepixel 12IR as the color filter 14C, an image of the infrared light IRalone can be obtained using the pixel data from the pixel IR.

The respective images thus generated are transmitted to an unshowndisplay unit, presented as a visible image to an operator, stored orsaved in a storage device such as a hard disk device or the like withoutany change, or transmitted to the other functional units as processeddata.

<Image Capturing Apparatus; Corresponding to CMOS>

FIGS. 5A and 5B are circuit diagrams of an image capturing apparatus inthe case of applying the layout of the color separation filtersillustrated in FIG. 2 to a CMOS solid state image capturing device (CMOSimage sensor).

Now, FIGS. 5A and 5B illustrate a configuration wherein the infraredlight IR is detected while separating the inside of a visible light bandinto the respective color components of R, G, and B, which is aconfiguration wherein blue light B, green light G, and red light Rwithin the visible light VL, and the infrared light IR are each detectedindependently, and substantially, which is a configuration whereinwithin one unit pixel matrix 12, pixels (photoelectric conversiondevices) 12B, 12G, and 12R are formed by wavelength, and also a pixel12IR having no wavelength separation configuration is included, and thepixel 12IR is used as a correction pixel as to the other pixels.

In the event of applying the layout of the color separation filtersillustrated in FIG. 2 to a CMOS, a configuration is employed wherein onecell amplifier is assigned to each of the pixels (photoelectricconversion devices) 12B, 12G, 12R, and 12IR within the unit pixel matrix12. Accordingly, this case takes a configuration such as FIG. 5A. Apixel signal is amplified by a cell amplifier, following which is outputthrough a noise cancel circuit or the like.

For example, a CMOS solid state image capturing device 201 has a pixelunit wherein multiple pixels including light receiving elements (oneexample of an electric charge generating unit) for outputting a signalcorresponding to the amount of incident light are arrayed with lines androws (i.e., in a two-dimensional matrix form), the signal output fromeach pixel is a voltage signal, and a CDS (Correlated Double Sampling)processing function unit, a digital conversion unit (ADC; Analog DigitalConverter), and so forth are provided in parallel with rows, i.e., forma so-called typical column type.

Specifically, as illustrated in FIGS. 5A and 5B, the CMOS solid stateimage capturing device 201 includes a pixel unit (image capturing unit)210 wherein multiple pixels 12 are arrayed with lines and rows, adriving control unit 207 which is provided at the outside of the pixelunit 210, a column processing unit 226, and an output circuit 228.

Note that the preceding and subsequent stages of the column processingunit 226 can be provided with an AGC (Automatic Gain Control) circuitand so forth having a signal amplifying function can be provided at thesame semiconductor region as the column processing unit 226 asnecessary. In the event of performing AGC at the preceding stage of thecolumn processing unit 226, this serves as an analog amplifier, and inthe event of performing AGC at the subsequent stage of the columnprocessing unit 226, this serves as a digital amplifier. Upon simplyamplifying n-bit digital data, gradation is sometimes deteriorated, andaccordingly, it can be conceived to preferably perform digitalconversion following amplification of analog data.

The driving control unit 207 includes a control circuit function forsequentially reading out the signals of the pixel unit 210. For example,the driving control unit 207 includes a horizontal scan circuit (rowscan circuit) 212 for controlling a row address and row scanning, avertical scan circuit (line scan circuit) 214 for controlling a lineaddress and line scanning, and a communication and timing control unit220 having functions such as an interface function as to the outside anda function for generating an internal clock, and so forth.

The horizontal scan circuit 212 has a function as a readout scan unitfor reading out a count value from the column processing unit 226. Theserespective components of the driving control unit 207 are integrallyformed at semiconductor regions such as single crystal silicon or thelike using the same technology as the semiconductor integration circuitmanufacturing technology along with the pixel unit 210, and areconfigured as a solid state image capturing device (image capturingdevice) serving as one example of a semiconductor system.

In FIGS. 5A and 5B, a part of lines and rows are omitted for the sake offacilitation of description, but actually, several tens through severalthousands of the pixels 12 are disposed at each of the lines and each ofthe rows. The pixels 12 are typically made up of the unit pixel matrices12 serving as light receiving elements (electric charge generatingunit), and intra-pixel amplifiers (cell amplifiers; pixel signalgenerating unit) 205 (205B, 205G, 205R, and 205IR by wavelength)including a semiconductor device for amplification (e.g., transistor).

Also, as can be understood from FIGS. 5A and 5B, the one unit pixelmatrix 12 is configured so as to detect the blue light B, green light G,red light R, and infrared light IR independently, and substantially,which is a configuration wherein the pixels 12B, 12G, 12R, and 12IR areformed by wavelength (color) within the one unit pixel matrix 12.

Now, as for the array of the color filters 14, for example, let us saythat the sequence in the X direction at the light receiving face side ofa silicon substrate 1_ω is blue, green, red, IR (correction pixel),blue, green, red, IR (correction pixel) and so on, and similarly, thesequence in the Y direction orthogonal to the X direction is blue,green, red, IR (correction pixel), blue, green, red, IR (correctionpixel), and so on. Also, it is effective to employ a color array inlight of deterioration in resolution by providing correction pixels(detailed description will be made later).

As for the intra-pixel amplifiers 205, amplifiers having a floatingdiffusion amplifier configuration are employed, for example. As for oneexample, an intra-pixel amplifier configured of four versatiletransistors serving as a CMOS sensor of a transistor for readoutselection serving as one example of an electric charge readout unit(transfer gate unit/readout gate unit), a reset transistor serving asone example of a reset gate unit, a transistor for vertical selection,and a transistor for amplification having a source followerconfiguration serving as one example of a detection element fordetecting electrical change of floating diffusion can be employed, as toan electric charge generating unit.

Alternatively, as described in Japanese Patent No. 2708455, anintra-pixel amplifier configured of three transistors of a transistorfor amplification connected to a drain line (DRN) for amplifying signalvoltage corresponding to signal electric charge generated by an electriccharge generating unit, a reset transistor for resetting the intra-pixelamplifier 205, and a transistor for readout selection (transfer gateunit) to be scanned by a vertical shift resistor via a transfer wiring(TRF) can be employed.

The pixels 12 are connected to the vertical scan circuit 214 via a linecontrol line 215 for line selection, and the column processing unit 226via a vertical signal line 219, respectively. Here, the line controlline 215 denotes the entire wiring entering to the pixel from thevertical scan circuit 214. As for one example, this line control line215 is disposed in the direction parallel with a long scatterer 3.

The horizontal scan circuit 212 and the vertical scan circuit 214 areconfigured, for example, including a shift register and a decoder so asto start address selection operation (scanning) in response to thecontrol signal given from the communication and timing control unit 220.Accordingly, the line control line 215 includes various types of pulsesignal for driving the pixels 12 (e.g., reset pulse RST, transfer pulseTRF, DRN control pulse DRN, etc.).

The communication and timing control unit 220 includes, though notillustrated in the drawing, the function block of a timing generator TG(one example of the readout address control device) for supplying clocknecessary for operation of each unit or a predetermined timing pulsesignal, and the function block of communication interface for receivinga master clock CLK0 via a terminal 220 a, also receiving data DATA forinstructing an operation mode and the like via a terminal 220 b, andfurther outputting data including the information of the CMOS solidstate image capturing device 201 via a terminal 220 c.

For example, the communication and timing control unit 220 outputs ahorizontal address signal to a horizontal decoder, and also a verticaladdress signal to a vertical decoder, and each of the decoders selectsthe corresponding line or row in response to the address signal, anddrives the pixels 12 and the column processing unit 226 via a drivingcircuit.

At this time, the pixels 12 are disposed in a two-dimensional matrixform, so it is desirable to realize speeding up of readout of a pixelsignal or pixel data by performing vertical scan reading for accessingand taking in the analog pixel signal in increments of line (in parallelwith rows), which is generated by the intra-pixel amplifier (pixelsignal generating unit) 205 and output in the row direction via thevertical signal line 219, and then performing horizontal scan readingfor accessing a pixel signal (e.g., digitalized pixel data) in the linedirection serving as the direction where vertical rows are arrayed, andreading out the pixel signal to the output side. It is needless to saythat random access for reading out the necessary information alone ofthe pixels 12 can be performed by directly specifying the address of thepixels 12 to intend to read regardless of scan reading.

Also, the communication and timing control unit 220 supplies a clockCLK1 having the same frequency as the master clock CLK0 to be input viathe terminal 220 a, a clock obtained by dividing the clock CLK1 intotwo, and a low-speed clock obtained by further dividing the clock CLK1to each unit within the device such as the horizontal scan circuit 212,vertical scan circuit 214, column processing unit 226, and so forth, forexample.

The vertical scan circuit 214 is a circuit for selecting a line of thepixel unit 210, and supplying a pulse necessary for line thereof. Forexample, the vertical scan circuit 214 includes a vertical decoder forstipulating a readout line in the vertical direction (selecting the lineof the pixel unit 210), and a vertical driving circuit for supplying apulse to the line control line 215 as to the pixels 12 on the readoutaddress (line direction) stipulated by the vertical decoder to drivevertical readout. Note that the vertical decoder selects a line forelectronic shutter in addition to a line for reading out a signal.

The horizontal scan circuit 212 is a circuit for sequentially selectingunshown column circuits within the column processing unit 226 in syncwith the low-speed clock CLK2, and guiding signal thereof to thehorizontal signal line (horizontal output line) 218. For example, thehorizontal scan circuit 212 includes a horizontal decoder forstipulating the readout row in the horizontal direction (selecting eachcolumn circuit within the column processing unit 226), and a horizontaldriving circuit for guiding each signal of the column processing unit226 to the horizontal signal line 218 using a selection switch 227 inaccordance with the readout address stipulated by the horizontaldecoder. Note that the number of the horizontal signal line 218 isdetermined by the number of bits n (n is a positive integer) which canbe handled by a column AD circuit for example, e.g., if the number ofbits is 10 (=n) bits, the ten horizontal signal lines 218 correspondingto this number of bits are disposed.

With the CMOS solid state image capturing device 201 thus configured,the pixel signal output from the pixels 12 is supplied to the columncircuit of the column processing unit 226 via the vertical signal line219 for each vertical row. Here, the signal electric charge accumulatedin the unit pixel matrix 12 (respective pixels 12B, 12G, 12R, and 12IR)is read out via the vertical signal line 219 of the same vertical row.

Each of the column circuits of the column processing unit 226 receivesthe pixel signal for the worth of one row, and processes signal thereof.For example, each of the column circuits has an ADC (Analog DigitalConverter) for converting an analog signal into, for example, 10-bitdigital data using, for example, the low-speed clock CLK2.

Also, devising a circuit configuration enables the pixel signal of thevoltage mode input via the vertical signal line 219 to be subjected toprocessing for taking difference between the signal level (noise level)immediately after pixel reset and a true signal level V_(sig)(corresponding to the amount of receiving light). Thus, noise signalcomponents such as fixed pattern noise (FPN) and reset noise can beremoved.

The analog pixel signal (or digital pixel data) processed at this columncircuit is propagated to the horizontal signal line 218 via a horizontalselection switch 217 to be driven by the horizontal selection signalfrom the horizontal scan circuit 212, and further is input to the outputcircuit 228. Note that 10 bits are one example, so the other number ofbits may be employed, such as less than 10 bits (e.g., 8 bits), greaterthan 10 bits (e.g., 14 bits), or the like.

According to such a configuration, the pixel signal regarding eachvertical row is sequentially output for each line from the pixel unit210 where the unit pixel matrix 12 (pixels 12B, 12G, 12R, and 12IR)serving as an electric charge generating unit is disposed in a matrixform. Consequently, one sheet of image, i.e., a frame imagecorresponding to the pixel unit 210 where light receiving elements aredisposed in a matrix form is illustrated with the group of pixel signalsof the entire pixel unit 210.

The output circuit 228 is a circuit corresponding to the outputamplifier 128 in the CCD solid state image capturing device 101, and theimage capturing signal processing unit 330 is provided at subsequentstage thereof, as with the CCD solid state image capturing device 101,as illustrated in FIG. 5B. An image switchover control signal from theimage switchover control unit 360 is input to the image capturing signalprocessing unit 330, as with the case of the CCD solid state imagecapturing device 101.

Thus, the image capturing data (each pixel data of R, G, B, and IR) ofthe primary colors of red (R), green (G), and blue (B), or the image ofa visible light band (so-called ordinary image) based on the image datafor the visible light VL can be obtained, and also an image relating tothe infrared light IR can be obtained by using the pixel data of theinfrared light IR.

<Signal Readout Method>

FIGS. 6A and 6B are diagrams for describing one example of the signalacquisition method in the case of employing an image sensor having aconfiguration for separating and obtaining a visible light image and aninfrared light image. The case of a CMOS configuration is shown, here.Note that FIG. 6A is a circuit diagram, and FIG. 6B is a signal timingchart.

Transfer gates 743 (R, G, and B) and 734IR are provided as to each ofphotoelectric conversion devices 732 (by color of R, G, and B in thecase of color, the following is the same) and 732IR provided in a lightreceiving portion by wavelength. Each of the photoelectric conversiondevices 732 (R, G, and B) and 732IR is connected to the intra-pixelamplifier 705 through each of the corresponding transfer gates 734 (R,G, and B) and 734IR, and an amplifying transistor 740 and a resettransistor 736. The amplifying transistor 740 is connected to a verticalsignal line 751 via a vertical selection transistor 742.

A pixel signal is output in accordance with each timing illustrated inFIG. 6B illustrating a reset state and a signal readout state. Here, ina state in which a selection pulse SEL is supplied to the verticalselection transistor 742 of the vertical line to be read out, prior tosupplying readout pulses T (R, G, and B) and TIR to the transfer gates734 (R, G, and B) and 734IR to read out each corresponding signalelectric charge, a floating diffusion 738 is reset by supplying a resetpulse RST to a reset transistor 736. Thus, a pixel signal can be readout in the sequence of the infrared light IR components, and the visiblelight VL components (components by color) (or vice versa thereof).

<Image Capturing Device; First Embodiment Using Dielectric Layered Film>

FIGS. 7A and 7B are diagrams for describing a first embodiment of thesolid state image capturing device 314. The solid state image capturingdevice 314 of this first embodiment has features in that the concept ofwavelength separation for subjecting electromagnetic waves to dispersionof light for each predetermined wavelength using a dielectric layeredfilm. Now, description will be made regarding subjecting light servingas one example of electromagnetic waves to dispersion for eachpredetermined wavelength as an example.

Specifically, the present embodiment utilizes the configuration proposedin Japanese Patent Application No. 2004-358139 by the present assignee,which is a spectral image sensor (spectral detection unit) having aconfiguration corresponding to wavelength separation utilizing adielectric layered film serving as a layered member having aconfiguration wherein multiple layers having predetermined thicknesswith adjacent layers having a different diffractive index are layered atthe incident face side where the electromagnetic waves of the solidstate image capturing device 314 are case, and properties wherein oflight (electromagnetic waves) to be cast in, wavelength components(infrared light IR components in the present example) which are otherthan the original detection target are reflected, and the remainingcomponents (visible light VL components in the present example) arepassed through. The basic configuration of the sensor may be any type ofa CCD type, CMOS type, and the other types.

Conversely, it can be conceived that the above properties possessed bythe layered member are properties for passing through the wavelengthcomponents (visible light VL components in the present example) of theoriginal detection target of incident light (electromagnetic waves), andthe remaining components (infrared light IR components in the presentexample) are reflected.

With the first embodiment, an image sensor having a spectral imagesensor configuration at the detection unit side of the visible light VLutilizing a dielectric layered film which is optimized for detection ofthe visible light VL. The infrared light IR is optically eliminatedusing a dielectric layered film, and photoelectrons alone of the visiblelight VL components alone which are cast into the detection unit of thevisible light VL are converted into electric signals. An arrangement ismade wherein spectral filters utilizing a dielectric layered film atvisible light detection pixels (specifically, each color pixel of R, G,and B) are integrally formed on the one image sensor without performingwavelength separation on the optical path, and spectral filtersutilizing a dielectric layered film are not formed at infrared lightdetection pixels, whereby a visible light image and an infrared lightimage can be independently simultaneously obtained. Thus, a visiblelight image and an infrared light image can be obtained independentlywith little influence being received from the infrared light IR.

<Concept of Wavelength Separation Utilizing a Dielectric Layered Film>

A dielectric layered film 1 is, as illustrated in FIG. 7A, a layeredmember having a configuration wherein the refractive index nj (j is apositive integer exceeding one; the following is the same) betweenadjacent layers differs (refractive index difference δn), and multiplelayers having a predetermined thickness dj are layered. Thus, asdescribed later, the dielectric layered film 1 can have properties forreflecting predetermined region components within electromagnetic waves,and passing through the remaining components.

With regard to how to count the number of layers of the respectivedielectric layers 1 _(—) j making up the dielectric layered film 1, thethick layers at both sides thereof (the n0'th layer 1_0 and the k'thlayer 1 _(—) k) are not counted as the number of layers, e.g., countingis performed from the first layer to the k'th layer in sequence.Substantially, the dielectric layered film 1 is made up of the basiclayers 1_1 through 1 _(—) n (n=5 in the drawing) excluding the thicklayers (the 0'th layer 1_0 and the k'th layer 1 _(—) k) at both sides.

Upon light being cast into the dielectric layered film 1 having such aconfiguration, reflectance (or transmittance) has certain dependency asto a wavelength λ due to interference at the dielectric layered film 1.Greater the reflectance difference δn of light is, stronger advantagethereof becomes.

Particularly, in the event that this dielectric layered film 1 has aperiodic configuration or a certain condition (e.g., the condition dthrough λ/4n of the thickness d of each layer), upon incident light L1such as white light or the like being cast in, only the reflectance ofthe light of a certain wavelength area (certain wavelength region light)is effectively raised to cause most components to change to reflectedlight components L2, i.e., transmittance is reduced, and also, thereflectance of light of wavelength regions other than that is reduced,thereby causing most components to change to transmitted lightcomponents L3, i.e., transmittance can be increased.

Here, the wavelength λ is the center wavelength of a certain wavelengtharea, and n is the refractive index of layer thereof. With the presentembodiment, a spectral filter 10 is realized by utilizing the wavelengthdependence of reflectance (or transmittance) due to this dielectriclayered film 1.

FIG. 7B illustrates a case wherein the infrared light IR and the visiblelight VL are subjected to dispersion of light. The dielectric layeredfilm 1 is formed so as to have high reflectance as to the infrared lightIR of the wavelength λ of an infrared region which is longer wavelengthside than the visible light VL (principally, longer wavelength side than780 nm), whereby the infrared light IR can be cut.

Note that the member (layered member) of the respective dielectriclayers 1 _(—) j makes up the dielectric layered film 1 using multiplelayers, and accordingly, the number of types of the member is at leasttwo types, and in the event of more than two layers, any of therespective dielectric layers 1 _(—) j may be made up of a differentlayer material, or two types (or greater than this) may be layeredalternately, or in an arbitrary sequence. Also, while the dielectriclayered film 1 is made up of basic first and second layer materials, apart thereof is may be substituted with a third (or greater than this)layer material. Specific description will be made below.

<<Design Method of Dielectric Layered Film; Example of Infrared LightCut>>

<Design Method of Thickness dj>

FIGS. 8 through 10 are diagrams for describing the basic concept of amethod for designing the dielectric layered film 1. Now, descriptionwill be made regarding a design example such that while the dielectriclayered film 1 is made up of basic first and second layer materials, theinfrared light IR is selectively reflected.

A configuration diagram thereof is illustrated in FIG. 8, and thedielectric layered film 1 employed for the present embodiment isconfigured so as to be sandwiched with the thick silicon oxide SiO2(hereinafter, referred to as SiO2) of both sides (hereinafter, lightincident side is referred to as the 0'th layer, and the opposite side isreferred to as the k'th layer), and layered with multiple dielectriclayers 1 _(—) j made up of the first and second layered members. Withthe example illustrated, let us say that as for the first and secondlayered members making up the dielectric layers 1 _(—) j, a commonmaterial is employed for any of both, two types of materials areemployed such that silicon nitride Si3N4 (hereinafter, referred to asSiN) is the first layered member, and silicon oxide SiO2 is the secondlayered member, and these are layered alternately. Also, with theconfiguration of the dielectric layered film 1, a case wherein above andbelow thereof have a sufficient thick silicon oxide SiO2 layer (d0=dk=∞)is assumed.

With such a dielectric layered film 1, reflectance can be effectivelyincreased by satisfying the following Expression (1).

[Expression 1]

dj=λ0/4nj  (1)

Here, dj (hereafter, j represents layer number) is the thickness of therespective dielectric layers 1 _(—) j making up the dielectric layeredfilm 1, nj is the refractive index of the respective dielectric layers 1_(—) j, and λ0 is the center wavelength of a reflected wavelength region(hereinafter, referred to as reflected center wavelength).

With regard to how to count the number of layers of the respectivedielectric layers 1 _(—) j making up the dielectric layered film 1, thethick silicon oxide of both sides thereof is not counted as the numberof layers, e.g., counting is performed in sequence from the first layertoward the k'th layer side, such as three layers for SiN layer/SiO2layer/SiN layer, and five layers for SiN layer/SiO2 layer/SiN layer/SiO2layer/SiN layer. FIGS. 5A and 5B illustrate a seven-layer configuration.

Also, let us say that the reflected center wavelength λ0 of the infraredlight IR serving as a reflected wavelength region is 900 nm, therefractive index nα of silicon nitride SiN making up the odd layers is2.03, the refractive index nβ of silicon oxide SiO2 making up the 0'th,even, and k'th layers is 1.46, and the refractive index difference δn is0.57.

Also, in accordance with the above Expression (1), let us say that thethickness dα (=d1, d3, and so on; j=odd number) of silicon nitride SiNlayer is 111 nm, and the thickness dβ (=d2, d4, and so on; j=evennumber) of silicon oxide SiO2 layer is 154 nm.

FIG. 9 illustrates the result of a reflectance R obtained by changingthe number of layers, and computing with the effectiveFresnel-coefficient method (reflectance spectrum diagram) regarding theconfiguration in FIG. 8 employing a common material, and thus, thenumber-of-layers dependent properties of reflectance spectrums can beunderstood.

As a result of FIG. 9, it can be understood that as the number of layersincreases, the reflectance R increases centered on the reflected centerwavelength λ0=900 nm of the infrared light IR. Further, it can beunderstood that a wavelength of 900 nm is thus selected as the reflectedcenter wavelength λ0, thereby almost dividing into the infrared light IRand the visible light VL. Here, it can be understood that employing fivelayers or more causes the reflectance R to be 0.5 or more, andparticularly, employing seven layers or more causes the reflectance R toexceed 0.7, which is preferable.

FIG. 10 is a reflectance spectrum diagram describing the fluctuationdependency (relation as to irregularities) of the thickness of thedielectric layers 1 _(—) j. Here, the case of seven layers is taken asan example, which illustrates the result (reflectance spectrum diagram)calculated by changing the thickness dj of each dielectric layer 1 _(—)j by +10%.

Conditional expression (1) is an ideal calculated value using theFresnel-coefficient method, but actually, the conditions of Expression(1) are loose and broad. For example, even if the error of the thicknessdj is ±10%, it can be understood from the calculation using theFresnel-coefficient method that reflectance can be effectivelyincreased.

For example, as illustrated in FIG. 9, even if there is a difference ofirregularities in the thickness dj, it can be understood that thereflectance R can be increased effectively. For example, it can beunderstood that the sufficient reflectance R can be obtained, such as0.5 or more at the reflected center wavelength λ0=900 nm of the infraredlight IR, and also even with the entire infrared light IR (principally,longer wavelength side than 780 nm), it can be understood thatreflection is strong. Accordingly, in actuality, even in light ofirregularities, a sufficient advantage for increasing the reflectanceeffectively can be obtained as long as the thickness dj of thedielectric layers 1 _(—) j is in a range of the following Expression(2).

[Expression 2]

0.9×λ0/4n≦dj≦1.1×λ0/4n  (2)

<Design Method of Reflected Center Wavelength λ0>

FIGS. 11A through 13 are diagrams describing the conditions of thereflected center wavelength λ0. The numerical conditions of thethickness dj depend on the bandwidth δλIR of the infrared reflectedregion of a spectrum. Such as the concept of reflection spectrums beingillustrated in FIG. 11A, in the event that the bandwidth δλIR of aninfrared reflected region is wide, unless the center wavelength λ0 isbrought to the long wavelength side, reflection at the visible light VLbecomes marked. Such as the concept of reflection spectrums beingillustrated in FIG. 11B, inversely, in the event that the bandwidth δλIRof an infrared reflected region is narrow, unless the center wavelengthλ0 is brought to the short wavelength side, reflection at an infraredregion close to the visible light VL will not occur. The wavelengthseparation performance between the visible light VL and the infraredlight IR is excellent.

Incidentally, according to the graph of the absorption spectrum ofsilicon Si illustrated in FIG. 1, of an infrared region, it can beunderstood that if the infrared light IR in a range of 0.78 μm≦λ≦0.95 μmcan be reflected, this is enough for an infrared cut advantage. This isbecause little light on the longer wavelength side from wavelength of0.95 μm is absorbed within the silicon Si, and subjected tophotoelectric conversion. Accordingly, the reflected center wavelengthλ0 should be selected so as reflect the infrared light IR having a wavelength region of a range of 0.78 μm≦λ≦0.95 μl.

Also, it can be conceived that even with the visible light VL, of a red(R) region, the light in a range of 640 through 780 nm is low invisibility, and accordingly, even if the light is reflected or notreflected, the light has particularly no influence on the performance ofan image capturing device. Accordingly, even if reflection occurs at awavelength region of 640 through 780 nm, there is no inconvenience.

Further, when the refractive index difference δn of the dielectriclayered film 1 is great, the bandwidth δλIR of an infrared reflectedregion becomes wide, and inversely, when the refractive index differenceδn is small, the bandwidth δλIR of an infrared reflected region becomesnarrow. Accordingly, the bandwidth δλIR of an infrared reflected regionbecomes narrow in the event of a SiN/SiO2 multi-layered film, andbecomes wide in the event of a Si/SiO2 multi-layered film.

Accordingly, in the event of a SiN/SiO2 multi-layered film (refractiveindex difference δn=0.57), it can be understood that the aboveconditions are satisfied as long as a range of 780 nm≦λ0≦950 nm based oncalculation between 780 nm illustrated in the reflection spectrumdiagram in FIG. 12 and the reflected center wavelength λ0 of 950 nm.Incidentally, FIG. 12 is a layered configuration such as later-describedFIG. 17, which is the results calculated by changing only the filmthickness dj of the dielectric layers 1 _(—) j so as to satisfy λ0=780nm and λ0=950 nm.

Similarly, in the event of a Si/SiO2 multi-layered film (refractiveindex difference ∈n=2.64), the above conditions are almost satisfied asa range of long as 900 nm≦λ0≦1100 nm, such as illustrated in thereflection spectrum diagram in FIG. 13.

As described above, with a combination of silicon nitride SiN or siliconSi and silicon oxide SiO2, the reflected center wavelength λ0 shouldsatisfy the following Expression (3-1). Preferably, it is desirable forthe reflected center wavelength λ0 to satisfy the following Expression(3-2). These mean that it is ideal to set the reflected centerwavelength λ0 to near 900 nm.

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 3} \rbrack & \; \\ \begin{matrix}{{780\mspace{14mu} {nm}} \leq {\lambda \; 0} \leq {1100\mspace{14mu} {nm}}} & ( {3\text{-}1} ) \\{{850\mspace{14mu} {nm}} \leq {\lambda \; 0} \leq {1000\mspace{14mu} {nm}}} & ( {3\text{-}2} )\end{matrix} \} & (3)\end{matrix}$

It is needless to say that the materials described above are only oneexample, and the advantage as described above is not always restrictedto a combination of silicon oxide SiO2 and silicon nitride SiN layer. Itis estimated by calculation that the same advantage can be obtained byselecting a material of which refractive index difference is 0.3 ormore, further preferably, 0.5 or more.

For example, a SiN film may have somewhat irregularities in constitutionthereof according to manufacturing conditions. Also, as for thedielectric layers 1 _(—) j making up the dielectric layered film 1, inaddition to silicon oxide SiO2 and silicon nitride SiN, oxide such asalumina Al2O3, zirconia ZrO2 (refractive index of 2.05), titanium oxideTiO2 (refractive index of 2.3 through 2.55), magnesium oxide MgO, zincoxide ZnO (refractive index of 2.1), and so forth, or a high polymermaterial such as polycarbonate PC (refractive index of 1.58), acrylicresin PMMA (refractive index of 1.49), and so forth, or a semiconductormaterial such as silicon carbide SiC (refractive index of 2.65),germanium Ge (refractive index of 4 through 5.5) can be employed.

The spectral filter 10 having features which are not included in thepast glass filter can be configured by employing a high polymermaterial. That is to say, a plastic filter can be provided, which excelsin light weight and durability (high temperature, high humidity, andimpact). <<Dielectric Layered Film Utilizing Demultiplexing ImageSensor>>

FIGS. 14 through 18 are diagrams describing one embodiment of a spectralimage sensor 11 appropriate for application to the solid state imagecapturing device 314 utilizing the dielectric layered film 1. Thisspectral image sensor 11 is configured by using the basic design methodof the spectral filter 10 utilizing the dielectric layered film 1. Now,description will be made regarding a design example of the spectralimage sensor 11 so as to cut the infrared light IR and receive thevisible light VL by forming the dielectric layered film 1 such asselectively reflecting the infrared light IR on a semiconductor devicelayer.

Note that with the basic configuration of the spectral image sensor 11,the spectral filter 10 is formed on the light receiving portion of asemiconductor device layer. This configuration alone provides thespectral image sensor 11 corresponding to single wavelengthdemultiplexing (i.e., for monochrome image capturing), but can become aspectral image sensor corresponding to color image capturing byproviding a predetermined color (e.g., any of R, G, and B) of a colorseparating filter corresponding to each light receiving portion of thespectral image sensor 11.

Here, when manufacturing the dielectric layered film 1 described withFIGS. 8 through 10 on a semiconductor device layer on which a detectiondevice such as silicon (Si) photodetector or the like is formed, havinggreater refractive index than that of the respective dielectric layers 1_(—) j making up the dielectric layered film 1, the distance from thesemiconductor device layer to the dielectric layered film 1, i.e., thethickness dk of silicon oxide SiO2 making up the k'th dielectric layer 1_(—) k is important.

This means, as illustrated in the configuration diagram in FIG. 14, thatthe spectrum of total reflected light LRtotal is changed due to crossprotection as to reflected light L4 from the surface of a siliconsubstrate 1_ω serving as the surface of a semiconductor device layer(photodetector, etc.) made up of silicon Si (refractive index of 4.1)for example.

FIG. 15 is a reflection spectrum diagram describing the fluctuationdependency of the thickness dk of a silicon oxide SiO2 layer making upthe dielectric layer 1 _(—) k of the total reflected light LRtotal.Here, the result computed by changing the thickness dk of the dielectriclayer 1 _(—) k regarding the dielectric layered film 1 having aseven-layer configuration illustrated in FIG. 8. With the respectivediagrams within FIG. 15, the horizontal axis is the wavelength λ (μm),and the vertical axis is the reflectance R.

As can be understood from the respective diagrams within FIG. 15, whenthe thickness dk is 0.154 μl, i.e., when the thickness dk is a valuesatisfying Conditional expression (1) as to the reflected centerwavelength λ0 of the infrared light IR, it can be understood that areflectance spectrum receives little influence, and strongly reflectsthe infrared light IR (wavelength λ≧780 nm). Conversely, with thespectrum of the thickness dk of 0.3 through 50 μm, it can be understoodthat another vibration occurs as compared with the reflectance spectrumof the thickness dk of ∞. Thus, it can be understood that there is awavelength region where infrared reflection deteriorates in a dip shape.

However, upon the thickness dk reaching 2.5 μm or more, the half valuewidth of an infrared dip becomes 30 nm or less, and particularly uponthe thickness dk reaching 5.0 μm or more, half value width thereofbecomes 20 nm or less, and accordingly, the half value width becomesnarrow sufficiently as to common broad natural light, so an averagedreflectance can be obtained. Further, with regard to the spectrum of thethickness dk of 0.3 through 1.0 μm, it can be understood that thereflectance at the visible light VL is also high. With these points inmind, it can be the that the thickness dk of around 0.154 μl, i.e., thevalue when satisfying Conditional expression (1) is preferably the mostappropriate.

FIG. 16 is a reflection spectrum diagram describing the fluctuationdependency of the thickness dk of a silicon oxide SiO2 layer making upthe dielectric layer 1 _(—) k, and particularly, illustrates the resultcalculated by changing the value of the thickness dk at around 0.154 μl.With the respective diagrams within FIG. 16, the horizontal axis is thewavelength λ (μm), and the vertical axis is the reflectance R.

As can be understood from this result, it can be understood that thereflectance at the infrared light VL can be suppressed in a range of thethickness dk=0.14 through 0.16 μm centered on the thickness dk=0.154 μmsatisfying Conditional expression (1).

In the light of the above description, the most appropriateconfiguration of the spectral image sensor 11 is, as illustrated in theconfiguration diagram in FIG. 17, substantially, a spectral image sensorhaving a dielectric layered film 1A of an eight-layer configurationincluding the dielectric layer 1 _(—) k of the k'th layer, and thecalculation results of reflection spectrum thereof is such as thereflection spectrum diagram illustrated in FIG. 18. In other words, thedielectric layered film 1A has a configuration wherein a layer made upof silicon oxide SiO2 serving as a second layered member is provided forthe worth of four cycles on the silicon substrate 1_ω.

<Modification Utilizing a Dielectric Layered Film>

FIGS. 19 through 31 are diagrams illustrating the modifications of thespectral filter 10 and the spectral image sensor 11 utilizing adielectric multi-layered film. The above configuration of the spectralfilter 10 shows the basic configuration utilizing the dielectric layeredfilm 1, and the other various modifications can be employed. Similarly,the above configuration of the spectral image sensor 11 shows the basicconfiguration wherein the spectral filter 10 utilizing the dielectriclayered film 1 is formed on a light receiving portion such as a CMOS,CCD, or the like, and the other various modifications can be employed.For example, though details are omitted, with regard to themodifications of the spectral filter 10 and the spectral image sensor11, various configurations can be employed such as proposed in JapanesePatent Application No. 2004-358139 by the present assignee.

For example, as with a first modification illustrated in FIG. 19, thereflection within a visible light region can be reduced by adding athird layer 1_γ (e.g., silicon nitride SiN layer) having an intermediaterefractive index as to the refractive index nk of the dielectric layer 1_(—) k of the k'th layer and the refractive index nω (=4.1) of thesilicon substrate 1_ω between the dielectric layer 1 _(—) k of the k'thlayer and the silicon substrate 1_ω, which is the first modification ofthe spectral image sensor 11.

Note that with the configuration illustrated in FIG. 19, when designingthe constants of the first layer through the seventh layer of thedielectric layered film 1, in response to the above modification, thereflected center wavelength λ0 of the infrared light IR is changed tonot 900 nm but 852 nm at further lower side, the thickness dα of siliconnitride SiN (=d1, d3, and so on; j=odd number) is set to 105 nm, and thethickness dβ of silicon oxide SiO2 layer (=d2, d4, and so on; j=evennumber) is set to 146 nm.

This is because the reflectance at the visible light is reduced, andalso the reflectance at the boundary of around 780 nm between thevisible light and the infrared light is also reduced simultaneously bynewly inserting a thin SiN layer (30 nm), so it is necessary tosupplement this deterioration worth by shifting the entirety to theshort wavelength side, i.e., effectively cut the infrared light aroundthe boundary. Of course, the reflected center wavelength λ0 of theinfrared light IR may be stayed at 900 nm.

Note that the third layered member added with the first modification isthe same as silicon nitride SiN serving as the first layered member, butthe other members may be employed as long as a member has greaterrefractive index than the silicon substrate 1_ω.

The spectral image sensor 11 having the first modification of thedielectric layered film 1 substantially includes a dielectric layeredfilm 1B of a nine-layer configuration in total of the dielectric layeredfilm 1 made up of seven layers, including two layers of the dielectriclayer 1 _(—) k of the k'th layer (silicon oxide SiO2) and the siliconnitride SiN layer 1_γ. The calculation result of reflectance spectrumsthereof is such as illustrated in FIG. 20.

Also, as a second modification illustrated in FIG. 21, a dark currentcan be further reduced by layering a fourth layer 1_δ (e.g., siliconoxide SiO2 layer) having smaller refractive index than the third layeredmember which is added in the first modification between the thirdlayered member and the silicon substrate 1_ω, which is the secondmodification of the spectral image sensor 11.

Specifically, a silicon oxide SiO2 layer 1_δserving as a fourth layer isemployed between the silicon nitride SiN layer 1_γ having the thicknessdγ which is the third layered member and the silicon substrate 1_ω, andthickness dδ thereof is set to 0.010 μm. The calculation result ofreflectance spectrums thereof is such as illustrated in FIG. 22.

Note that the fourth layered member added with the second modificationis the same as silicon oxide SiO2 serving as the second layered member,but the other members may be employed as long as a member has smallerrefractive index than the third layered member (silicon nitride SiN inthe present example).

The spectral image sensor 11 having the second modification of thedielectric layered film 1 substantially includes a dielectric layeredfilm 1C of a ten-layer configuration in total of the dielectric layeredfilm 1 made up of seven layers, including three layers of the dielectriclayer 1 _(—) k of the k'th layer (silicon oxide SiO2), the siliconnitride SiN layer 1_γ, and the silicon oxide SiO2 layer 1_δ. In otherwords, the dielectric layered film 1C has a configuration wherein alayer made up of silicon oxide SiO2 serving as the second layered memberis provided for the worth of five cycles on the silicon substrate 1_δ.

There is a difference between the first modification and the secondmodification regarding whether or not the silicon oxide SiO2 layer 1_δexists, but as can be understood from FIGS. 20 and 22, either of thesecan sufficiently reduce the reflectance at the visible light VL. Also,an advantage wherein a dark current can be reduced can be obtained byadding the silicon oxide SiO2 layer 1_δ, such as the secondmodification. Note that in order not to reduce the advantage obtained byadding the silicon nitride SiN layer 1_γ by adding the silicon oxideSiO2 layer 1_δ, it is desirable to set the relation between boththicknesses to dδ<<dγ.

Thus, the reflection at the visible light VL can be suppressed by addingthe thin silicon nitride SiN layer 1_γ serving as a member anintermediate refractive index nγ (=nSiN) as to the refractive index nk(=nSiO2) and the refractive index nω (=nSi) between the silicon oxideSiO2 of the k'th layer and the silicon substrate 1_ω as an intermediatelayer.

Also, the number of layers of the dielectric layers 1 _(—) j making upthe dielectric layered film 1 can be reduced by adding a fifth layer 1_η(e.g., silicon Si layer having a thickness dη of 61 nm and refractiveindex of 4.1 which is higher than silicon nitride SiN and silicon oxideSiO2) having greater refractive index than the basic first and secondlayered members making up the dielectric layered film 1 to the inside ofthe dielectric layered film 1 of the spectral image sensor 11, such as athird modification illustrated in FIG. 23.

For example, with the example illustrated in the configuration diagramin FIG. 23, only one layer of a silicon Si layer having a refractiveindex of 4.1, which is higher than silicon nitride SiN and silicon oxideSiO2, and a thickness dη of 61 nm is added as the fifth layered memberinstead of silicon nitride SiN (instead of a dielectric layer 1_3 of theintermediate third layer).

Note that in FIG. 23, when designing the constants of the respectivelayers of the dielectric layered film 1, the reflected center wavelengthλ0 of the infrared light IR is set to 900 nm, the thickness dα (=d1, d3,and so on; j=odd number) of a silicon nitride SiN layer is set to 111nm, the thickness dβ (=d2, d4, and so on; j=even number) of a siliconoxide SiO2 layer is set to 154 nm, and only one layer of a silicon Silayer of a thickness dη of 55 nm is added as the fifth layered memberinstead of a silicon nitride SiN layer. The calculation result ofreflectance spectrums thereof is such as the reflectance spectrumdiagram illustrated in FIG. 24.

The fifth layered member added in the third modification is the same asthe silicon substrate 1_ω serving as a semiconductor device layer, butthe other members may be employed as the fifth layered member as long asa member has greater refractive index than the dielectric layer 1 _(—) jother than the fifth layered member serving as the dielectric layeredfilm 1.

Note that when manufacturing a dielectric layered film 1D on asemiconductor device layer (silicon substrate 1_ω), the distance fromthe semiconductor device layer to the dielectric layered film 1D, i.e.,the thickness dk of a silicon oxide SiO2 layer serving as the dielectriclayer 1 _(—) k of the k'th layer is important.

This means, as illustrated in the configuration diagram in FIG. 23, thatthe spectrum of total reflected light LRtotal is changed due to crossprotection as to reflected light LR from the surface of the siliconsubstrate 1_ω serving as the surface of a semiconductor device layer(photodetector, etc.) made up of silicon Si (refractive index of 4.1)for example.

FIG. 25 is a reflectance spectrum diagram describing the fluctuationdependency of the thickness dk of the silicon oxide SiO2 layer servingas the dielectric layer 1 _(—) k of the total reflected light LRtotalregarding the dielectric layered film 1D of a five-layer configuration.With the respective diagrams within FIG. 25, the horizontal axis is thewavelength λ (μm), and the vertical axis is the reflectance R.

As can be understood from the respective diagrams within FIG. 25, whenthe thickness dk is 0.15 μm, i.e., when the thickness dk is a value0.154 μm satisfying Conditional expression (1) as to the reflectedcenter wavelength λ0 of the infrared light IR, it can be understood thata reflectance spectrum receives little influence, and strongly reflectsthe infrared light IR (wavelength λ≧780 nm). Conversely, with thespectrum of the thickness dk of 0.3 through 50 μm, it can be understoodthat another vibration occurs as compared with the reflectance spectrumof the thickness dk of ∞. Thus, it can be understood that there is awavelength region where infrared reflection deteriorates in a dip shape.

Also, when reducing the number of layers in the third modification, thenumber of layers can be further reduced by adding multiple fifth layers1_η (e.g., silicon Si layers having a thickness dη of 61 nm andrefractive index of 4.1 which is higher than silicon nitride SiN andsilicon oxide SiO2) having greater refractive index than the basic firstand second layered members making up the dielectric layered film 1 tothe inside of the dielectric layered film 1 of the spectral image sensor11, such as the fourth modification illustrated in FIGS. 26 and 28. Thisarrangement has features in that the refractive index difference of theadjacent layers in the basic layers is great.

For example, with the example illustrated in the configuration diagramin FIG. 26 (fourth modification; first thereof), a dielectric layeredfilm 1E is configured of the basic layers as a three-layerconfiguration, two silicon Si layers having a high refractive index of4.1, which is higher than silicon nitride SiN and silicon oxide SiO2,and a thickness dη of 61 nm are provided as fifth layered membersinstead of silicon nitride SiN. In other words, the dielectric layeredfilm 1E has a configuration wherein a layer made up of silicon oxideSiO2 serving as a second layered member is provided for the worth of twocycles on the silicon substrate 1_ω. The calculation result ofreflectance spectrums thereof is such as the reflectance spectrumdiagram illustrated in FIG. 27.

Note that when designing the constants of the respective layers of thedielectric layered film 1, the reflected center wavelength λ0 of theinfrared light IR is set to 1000 nm, the thickness dη (=d1, d3) of thesilicon Si layers made up of the fifth layered members is set to 61 nm,and the thickness dβ (=d2) and dk of the silicon oxide layers is set to171 nm.

Also, with the example illustrated in the configuration diagram in FIG.28 (fourth modification; second thereof), a dielectric layered film 1Eis configured of the basic layers as a five-layer configuration, threesilicon Si layers having a high refractive index of 4.1, which is higherthan silicon nitride SiN and silicon oxide SiO2, and a thickness dry of61 nm are provided as fifth layered members instead of silicon nitrideSiN. The calculation result of reflectance spectrums thereof is such asthe reflectance spectrum diagram illustrated in FIG. 29. As can beunderstood from comparison as to FIG. 27, it can be understood that thereflectance at an infrared light region can approximate 1.0 byincreasing the number of the fifth layered members.

Also, with the third or fourth modification, the first modification isapplied to the spectral image sensor 11 simultaneously, wherebyreflection within a visible light region can be reduced as well asreduction of the number of layers, such as the fifth modificationillustrated in FIG. 30. Particularly, though the reflectance at blue Bcomponents (wavelength of around 420 nm) and green G components(wavelength of around 520 nm) somewhat increases, the reflectance at redR components (wavelength of around 600 nm) can be sufficiently reduced,which is suitable for separation from the infrared light IR.

For example, with the configuration illustrated in FIG. 30, thethickness dγ is configured of layering relatively thin silicon nitrideSiN layer 1_γ as the third layered member between the silicon oxide SiO2at the k'th layer and the silicon substrate 1_ω. Here, the thickness dγis set to 0.030 μm. The calculation result of reflectance spectrumsthereof is such as the reflectance spectrum diagram illustrated in FIG.31. The spectral image sensor 11 having the dielectric layered film 1 ofthis modification substantially includes a dielectric layered film 1F ofa five-layer configuration in total of the dielectric layered film 1made up of three layers, including two layers of the dielectric layer 1_(—) k of the k'th layer (silicon oxide SiO2 layer) and the siliconnitride SiN layer 1_γ.

Note that the third layered member added with the present fifthmodification is the same as silicon nitride SiN serving as the firstlayered member, but the other members may be employed as long as amember has greater refractive index than the silicon substrate 1_ω.

Note that though illustration is omitted, with the third or fourthmodification, the first and second modifications are applied to thespectral image sensor 11 simultaneously, whereby reflection within avisible light region can be reduced, and a dark current can be reducedas well as reduction of the number of layers.

With the above description, the spectral image sensor 11 is configuredusing the spectral filter 10 which utilizes the dielectric layered film1, but the spectral image sensor 11 is not restricted to this as long asa member having properties for reflecting a predetermined wavelengthregion components within electromagnetic waves, and passing theremaining components is provided at the incident face side where theelectromagnetic waves of a detection unit are cast.

For example, a spectral filter can be formed even utilizing a singlelayer film of a predetermined thickness regardless of the dielectriclayered film 1. This is because, upon the film thickness of a singlelayer film being changed, an advantage is obtained wherein predeterminedwavelength region components within electromagnetic waves are reflected,and the remaining components are passed through.

<<Specific Example of Manufacturing Process>>

FIG. 32 is a diagram illustrating a specific process example formanufacturing a spectral image sensor having a sensor configurationutilizing the layered film described with the above embodiment. ThisFIG. 32 is a manufacturing process example of a spectral image sensorincluding a light receiving portion for the infrared light IR and alight receiving portion for the visible light VL.

When manufacturing this configuration, first of all, a common CCD, and acircuit having a CMOS configuration are formed (FIG. 32A). Subsequently,a SiO2 film and SiN are sequentially layered on a Si photodiode usingthe CVD (Chemical Vapor Deposition) method or the like (FIG. 32B).

Subsequently, an opening portion which reaches the light receivingportion for the infrared light IR, and the SiO2 film at the lowest layeris provided, for example, by subjecting only one of four pixels toetching using lithography technology, the RIE (Reactive Ion Etching)method, or the like (FIG. 32E). The light receiving portion for theinfrared light IR is also used as a correction pixel as to the othercolor pixels for visible light color image capturing.

Subsequently, a SiO2 film is layered and smoothed on the dielectriclayered film 1 of which a part is provided with an opening portionusing, for example, the CVD method or the like again for the sake ofprotecting the dielectric layered film 1 and the like (FIG. 32F). Ofcourse, this process is not indispensable.

Note that at this time, a photoresistor where an opening portion isprovided in the light receiving portion for the infrared light IR may beemployed so as not to subject three pixels for the visible light VL (forR, G, and B components) to etching (FIGS. 32C and 32D). In this case, itis necessary to remove the photoresistor prior to layering the SiO2 filmon the dielectric layered film 1 (FIGS. 32D and 32E).

Also, though an illustration is omitted, a color filter and a micro lensare formed further on the above thereof so as to correspond to a pixel.At this time, for example, let us say that a black filter is disposed inthe light receiving portion for the infrared light IR, and primary-colorfilters are disposed in the detection unit for visible light, and thus,the pixel of the black filter receives infrared light, and the otherthree color pixels receive the three primary colors of red, green, andblue of visible light.

Thus, the dielectric multi-layered film of a SiN layer and a SiO2 layerare formed on the detection unit of the three primary-color visiblelight pixels, but this dielectric multi-layered film is not formed onthe detection unit of the black filter pixel. Three primary-colorvisible light images and an infrared light image can be capturedsimultaneously by employing the image capturing device manufactured withsuch a configuration.

Further, in the event that a little infrared light IR is filtered in aphotoelectric conversion device for the visible light VL (photodiode orthe like), weak infrared light cut filters may be inserted entirely. Forexample, the infrared light IR is condensed at the photoelectricconversion device for the infrared light IR (photodiode or the like)even if the visible light VL is cut to an almost satisfactory level byinserting an infrared light cut filter of 50% or less, wherebysufficient sensitivity can be obtained.

Note that with such a manufacturing process, up to near the Si substratesurface is subjected to etching, i.e., an opening portion which reachesthe light receiving portion for the infrared light IR, and the SiO2 filmat the lowest layer is provided (FIG. 32E), so damage due to etchingsometimes results in a problem. In this case, damage can be reduced byincreasing the thickness d of the SiO2 layer immediately above the Sisubstrate.

Here, upon the dk reaching 2.5 μm or more, the half value width of thedip in an infrared light region of reflectance spectrums becomes narrowas illustrated in FIG. 15, and thus, the reflectance averaged as tocommon broad natural light can be obtained, and consequently, reflectionof infrared light can be performed. Accordingly, it is desirable to setthe thickness dk of the dielectric layer 1 _(—) k of the k'th layer to2.5 μm or more. It is further desirable to set this to the thickness of5 μm or more.

Also, in the event that metal wiring for the photodiode, intra-pixelamplifier, or the like to be formed on the silicon substrate 1_ω, i.e.,a wiring layer serving as a signal line for reading out a pixel signalserving as a unit signal output from the intra-pixel amplifier servingas a unit signal generating unit from an image capturing unit (detectionregion) is formed on the immediately above the silicon substrate 1_ω,the dielectric layered film 1 is formed not immediately above thesilicon substrate 1_ω but at the position isolated to some extent on thesilicon substrate 1_ω, i.e., the dielectric layered film 1 is formedabove the metal wiring, whereby the process can be readily performed,and an advantage wherein cost is kept low can be obtained. Detaileddescription will be made later, but a certain degree of good results canbe obtained by increasing the number of layers making up the dielectriclayered film 1. Description will be made below regarding a spectralimage sensor which considers metal wiring.

<<Demultiplexing Image Sensor Utilizing Dielectric Layered Film; SixthModification>>

FIGS. 33 through 39 are diagrams describing a sixth modification of aspectral image sensor 11 corresponding to single wavelength divisionutilizing the dielectric layered film 1. The sixth modification hasfeatures in that taking the technique described with FIGS. 14 through 18as the basic, in light of metal wiring, the dielectric layered film 1 isformed on the silicon substrate 1_ω, which is above the siliconsubstrate 1_ω with some degree of distance, integrally with thedetection unit such as a photodiode or the like.

For example, as illustrated in FIG. 33, in light of a CMOSconfiguration, in the event that one wiring layer is provided on asemiconductor device layer on which a detection unit such as aphotodiode or the like is formed, and thickness thereof is around 0.7μm, when a multi-layered film configuration is integrally formedgenerally 0.7 μm above than the silicon substrate 1_ω on which aphotodiode and the like is formed, the dielectric layered film 1 shouldbe formed following the process of the wiring layer of the first layer.Thus, a wiring layer can be provided within the k'th layer having thethickness dk of generally equal to 0.7 μm.

Also, as illustrated in FIG. 34, in the event that three wiring layersare provided on a semiconductor device layer, and total thicknessthereof is around 3.2 μm, when integrally forming a multi-layered filmconfiguration generally above 3.2 μm than the silicon substrate 1_ω onwhich a photodiode and so forth are formed, the dielectric layered film1 should be formed following the process of the wiring layer of thethird layer which is the uppermost layer. Thus, a wiring layer can beprovided within the k'th layer having the thickness dk equal to 3.2 μm.

The reason why “generally 3.2 μm” has been described here is that asillustrated in the drawing, with the present example, a SiO2 layer (δlayer) having a thickness of around 10 nm is provided on the siliconsubstrate 1_ω, and a SiN layer (γ layer) having a thickness of around 65nm is provided on the above thereof, and “3.2 μm” means the thickness ofthe k layer except for these γ and δ layers.

The color filters 14, micro lens, and so forth should be formedfollowing this dielectric layered film 1 being formed.

FIGS. 35 and 36 are diagrams illustrating the concept of the layeredconfiguration of such a spectral image sensor 11. Now, the configurationof the second modification illustrated in FIG. 21 is utilized wherein inaddition to the basic layers 1_1 through 1 _(—) n, the third layer 1_γand the fourth layer 1_δ are provided between the dielectric layer 1_(—) k of the k'th layer and the silicon substrate 1_ω. Also, as withthe second modification, the reflected center wavelength λ0 of theinfrared light IR is set to 852 nm.

For example, in FIG. 35, while the seven-layer configuration illustratedin FIG. 21 is employed as the basic layer, the dielectric layered film1C having three layers of the dielectric layer 1 _(—) k of the k'thlayer (silicon oxide SiO2 layer), the silicon nitride SiN layer 1_γ, andthe silicon oxide SiO2 layer 1_δ is taken as the base, the thickness ofthe dielectric layer 1 _(—) k of the k'th layer is set to 700 nm. Also,the relatively thin silicon nitride SiN layer 1_γ having a thickness dγof 65 nm or 100 nm is layered as a third layered member between thesilicon oxide SiO2 of the k'th layer and the silicon substrate 1_ω, andfurther, the silicon oxide SiO2 layer 1_δ serving as a fourth layerhaving smaller refractive index than the third layered member is layeredwith a thickness dδ of 10 nm between this added third layered member andthe silicon substrate 1_ω, thereby forming the dielectric layered film1C.

Also, in FIG. 36, while the dielectric layered film 1 serving as thebase has a nine-layer configuration, the thickness of the dielectriclayer 1 _(—) k of the k'th layer is set to 700 nm or 3.2 μm. Also, therelatively thin silicon nitride SiN layer 1_γ having a thickness dγ of65 nm is layered as a third layered member between the silicon oxideSiO2 of the k'th layer and the silicon substrate 1_ω, and further, thesilicon oxide SiO2 layer 1_δ serving as a fourth layer having smallerrefractive index than the third layered member is layered with athickness dδ of 10 nm; thereby forming the dielectric layered film 1C.

The calculation results of these reflectance spectrums is such asillustrated in FIGS. 37 through 39. As can be understood from FIGS. 33and 34, the dielectric layered film 1 is formed 0.7 μm or 3.2 μm abovethe silicon substrate 1_ω, thereby facilitating the wiring process. Notethat accurately, the SiO2 layer serving as the fourth layer and the SiNlayer serving as the third layer are provided immediately above thesilicon substrate 1_ω in that order having thickness of 10 nm and 65 nm(or 100 nm) respectively, and accordingly, the position where thedielectric layered film 1 is formed is above those in total.

Here, with the dielectric layered film 1 having the SiN film and theSiO2 film, the case of seven layers and the case of nine layers havebeen shown, but it can be understood from FIG. 37 that if the siliconnitride SiN layer 1_γ is excessively thickened, the dip in an infraredlight reflected region is great, and consequently, reflection is greatlydeteriorated. In addition, if the thickness dγ of the SiN layer servingas the third layered member is thick, reflection in a visible lightregion is raised. This can be conceived that as described with thesecond modification, the third layered member, which is provided as anintermediate layer, aims at reducing the reflection in a visible lightregion, and if the thickness dy of the dielectric layer 1_γ which isprovided as an intermediate layer is thin, sufficient room is provided,otherwise, little room is provided.

Also, as can be understood from FIG. 38, it can be understood that uponthe number of multi-layered configuration being increased up to 9layers, the reflectance R in an infrared light region exceeds 0.9, thereflection performance at an infrared light region can be furtherimproved as compared with the case of 7 layers. Also, as can beunderstood from FIG. 39, it can be understood that with a 7-layerconfiguration wherein the thickness dk of the dielectric layer 1 _(—) kof the k'th layer is 3.2 μm, the dips in an infrared light reflectedregion are great, and consequently, reflection is greatly deteriorated.However, even with regard to this case, it can be understood that uponthe number of layers being increased up to 9 layers, these dips becomesmall, the reflection performance at an infrared light region can befurther improved.

Thus, forming the dielectric layered film 1 following the past wiringprocess being performed facilitates manufacturing, and consequently,eliminates the necessity of the study of a new process, and isadvantageous cost-wise. That is to say, manufacturing the CMOSconfiguration such as FIGS. 33 and 34 facilitates the process, and alsoan effective advantage can be obtained. If a wiring process is performedfollowing the dielectric layered film 1 being formed, a great differenceis provided such that it becomes difficult in a process manner toperform removal of the dielectric layered film 1 and so forth.

<Image Capturing Device; Second Embodiment Using Diffraction Grating>

FIG. 40 is a diagram for describing a second embodiment of the solidstate image capturing device 314. The solid state image capturing device314 of this second embodiment has features in that the concept ofwavelength separation for spectrally separating electromagnetic wavesfor each predetermined wavelength using diffraction grating. Now,description will be made regarding spectrally separating light servingas one example of electromagnetic waves for each predeterminedwavelength.

Specifically, the present embodiment is an embodiment utilizing theconfiguration which the present assignee has proposed in Japanese PatentApplication No. 2004-250049, a diffraction grating 501 is, asillustrated in FIG. 40A, has a configuration wherein scatterers 502 madeup of a shielding member for cutting off (shielding in the case oflight) electromagnetic waves (e.g., light) are arrayed periodically,upon incident light L1 being cast in, diffracted waves L2 are generatedby light being scattered at each of the scatterers 502. Also,interference occurs among the diffracted waves L2 by the scatterers 502including a great deal of the diffracted waves L2 periodically.

Thus, as illustrated in FIG. 40B, light intensity becomes strong at aplace where the phases of the respective diffracted waves L2 arematched, and inversely, becomes weak at a place where the phases areshifted for the worth of a half wavelength. Consequently, interferencefringes are generated straddle between the surface of a Si (silicon)substrate 509 and the inside of the Si substrate.

The present embodiment realizes a spectral image sensor by utilizingthat the pattern of this interference fringe exhibits wavelengthdispersibility which is changed by a wavelength λ.

<<Basic Configuration of Demultiplexing Image Sensor UtilizingDiffraction Grating>>

FIG. 41 is a conceptual diagram describing the basic configuration of ademultiplexing image sensor (spectral image sensor) utilizing adiffraction grating. FIG. 42 is a diagram enlarging and illustrating onephotodiode group 512 of the spectral image sensor 511 illustrated inFIG. 41.

Here, the photodiode group 512 is equivalent to one photodiodecorresponding to one pixel in the past image capturing device. Thepresent embodiment has features in that multiple photoelectricconversion devices by wavelength (by color) for detecting differentwavelengths (color components) spectrally separated utilizing adiffraction advantage caused by the electromagnetic waves passingthrough an opening portion are provided within the photodiode group 512,thereby realizing high resolution and high pixilation. The configurationaccording to the present embodiment is a configuration wherein theentire color components for color separation are assigned to onephotodiode group 512 making up one pixel, and is different from aconfiguration wherein any of color components for color separation isassigned to a photoelectric conversion device by color making up onepixel.

The spectral image sensor 511 wherein the photodiode group 512 made upof multiple photoelectric conversion devices illustrated in FIG. 41 arearrayed has features in that electromagnetic waves are divided intomultiple wavelength components utilizing a diffraction advantage, awavelength dispersion portion for having the respective wavelengthcomponents cast into adjacent different positions within the incidentface is made up of scatterers 503 (first scatterer), slits 505 (secondscatterer), and scatterers 507 (third scatterer).

Specifically, the scatterers 503 made up of long-shaped (linear orrod-shaped) opening portions 503 a for passing through electromagneticwaves (e.g., light) at the incident light L1 side, and light shieldingportions 503 b in a long shape (line shape or rod shape) which areportions for cutting off (shielding in the case of light)electromagnetic waves (e.g., light), which are alternately arrayed, aredisposed such that the light shielding portions 503 b are in parallel,periodically, and arrayed in a predetermined direction (e.g., horizontaldirection; X direction in the drawing) within the light receiving face.The horizontal direction (X direction in the drawing) corresponds to thereadout direction of a pixel signal from the photoelectric conversiondevice by color included in the photodiode group 512.

Here, the term “linear” means a shape of which cross-sectional area isboundlessly close to zero as to the wavelength of light, and the term“rod-shaped” means a shape of which cross-sectional area is limited, butboth are common in that both are in a slender shape, i.e., a long shape.

Also, with the spectral image sensor 511, a slit-shaped light scatterer(hereinafter, referred to as slits) 505 making up the principal portionof the diffraction grating 501 is provided such that a long openingportion (gap of slit) 505 a for passing through electromagnetic waves(e.g., light), and a light shielding portion 505 b which is a portionfor cutting off (shielding in the case of light) electromagnetic waves(e.g., light) surrounding the opening portion 505 a are disposed on thefringe where the diffracted waves L2 between the scatterer 503 and theSi substrate 509 are gathered.

With the slits 505, the cycle of the opening portions 505 a (theinterval of the adjacent opening portions 505 a) and the cycle of thescatterers 503 (the interval of the light shielding portions 503 b ofthe adjacent scatterers 503; the interval of the opening portions of thescatterer 503) are equal, and also these are disposed in parallel. Inaddition, the opening portion 505 a is disposed at the position(vertical as to the photodiode group 512 surface) on a generally centerline CL0 passing through the center point between the adjacentscatterers 503.

Further, with the spectral image sensor 511, the scatterers 507 whereinthe different long light shielding portions 507 b are disposedperiodically in parallel while sandwiching the opening portion 507 a isprovided between the slits 505 and the photodiode group 512. Thescatterers 507 are set such that the cycle of the scatterers 507 (theinterval of the adjacent light shielding portions 507 b), and the cycleof the scatterers 503 (the interval of the adjacent light shieldingportions 503 b of the scatterers 503) or the cycle of the openingportions 505 a of the slits 505 (the interval of the adjacent openingportions 505 a of the slits 505) become equal. In addition, the lightshielding portions 507 b of the scatterers 507 are arranged so as to bedisposed at the position (vertical as to the photodiode group 512surface) on a generally center line CL0 passing through the center pointbetween the scatterers 503.

With the spectral image sensor 511 having such a configuration, therespective scatterers 503 are disposed periodically, and places wherethe diffracted waves L2 are condensed appear such as FIG. 41.Particularly, in the event that the respective scatterers 503 aredisposed in the horizontal direction periodically, and also in parallelwith an equal interval, the diffracted waves L2 are gathered together onthe center line CL0 between the respective scatterers 503.

Also, the diffracted waves L2 are condensed in a long shape, or theinterference fringe (light intensity distribution) between thediffracted waves L2 becomes a long shape by employing the long-shapedscatterers 503. Accordingly, upon considering the device configuration,the photoelectric conversion devices (photodetectors) to be providedwithin the photodiode group 512 can be configured in a long shape,thereby providing an advantage wherein it is easy to design.

In addition, FIG. 41 illustrates the basic configuration of the spectralimage sensor paying attention to a condensing advantage, butsubsequently, as the diffracted waves L2 are further advanced, theincident light L1 is spectrally separated by the advantage of the slits505 and another one set of scatterers 507 as illustrated in FIG. 42.

Accordingly, condensing and spectral separation can be realized with acombination of these two advantages. At this time, the slits 505 aredisposed between the scatterers 503 and the photodiode group 512 whichare the peripheries where the diffracted waves L2 gather together suchthat the long-shaped opening portions 505 a (gap of slits) areperiodical in the horizontal direction and also in parallel, whereby theincident light L1 can be spectrally separated in a well controlledmanner, such as blue light L3-B, green light L3-G, red light L3-R, andinfrared light L3-IR (collectively referred to spectral components L3).

With the spectral image sensor 511 illustrated in FIG. 41 in which thephotodiode group 512 made up of multiple photoelectric conversiondevices are arrayed, a long-shaped (linear or rod-shaped) scatterers 503are disposed so as to array in a predetermined direction (e.g.,horizontal direction; X direction in the drawing) within the lightreceiving face in parallel periodically. The horizontal direction (Xdirection in the drawing) corresponds to the readout direction of apixel signal from the photoelectric conversion device by color includedin the photodiode group 512.

Also, particularly, if the cycle of the opening portions 505 a of theslits 505 (the interval of the adjacent opening portions 505 a) and thecycle of the scatterers 503 (the interval of the adjacent lightshielding portions 503 b) are equal, and also these are disposed inparallel, periodicity and symmetry improves in the entire configuration(particularly, relation between the scatterers 503 and the slits 505),and the interference properties of light improves. Consequently,spectral properties improve.

Also, the light (diffracted waves L2) diffracted at the scatterers 503is condensed around center point thereof (the center between the lightshielding portions 503 b). Thus, the opening portions 505 a of the slits505 are arranged so as to be disposed at the position (vertical as tothe photodiode group 512 surface) generally on the center line CL0passing through the center point between the scatterers 503, therebyperforming spectral separation effectively.

Also, it has been found that the visible light (blue light L3-B, greenlight L3-G, and red light L3-R) and the infrared light L3-IR can beseparated spectrally by setting the slit width Ds to 800 nm±300 nm.

Further, it has been also found that another one set of scatterers 507may be disposed or may not be disposed between the slits 505 and thephotodiode group 512 on the center line CL0 of the opening portions 505a of the slits 505 in some cases.

Here, it has turned out by the optical field computation of the FDTD(Finite Difference Time Domain) method in accordance with the Maxwellequation that another set of scatterers 507 is disposed on the centerline CL0 of the opening portions 505 a of the slits 505 and also betweenthe slits 505 and the photodiode group 512, and thus, as illustrated inthe enlarged view in FIG. 42, the light at the long wavelength side(green light through red light) is condensed at a position apart fromthe scatterers 507 by diffraction, but the light at the short wavelengthside (particularly, blue light) is condensed on the center line CL0 ofthe photodiode group 512 at the rear side of scatterer thereof.

Here, the diffracted waves L2 are condensed in a long shape, or theinterference fringe (light intensity distribution) between thediffracted waves L2 becomes a long shape by employing the long-shapedscatterers 507. Accordingly, upon considering the device configuration,the photoelectric conversion devices (photodetectors) to be providedwithin the photodiode group 512 can be configured in a long shape,thereby providing an advantage wherein it is easy to design. Also, thescatterers 507 are disposed periodically in parallel, thereby improvingperiodicity and symmetry in the entire configuration, and also improvingthe interference properties of light. Consequently, spectral propertiesimprove. A combination of both provides a configuration wherein spectralproperties are excellent, and it is easy to design.

Also, if the cycle of the scatterers 503 (the interval of the adjacentlight shielding portions 503 b) and the cycle of the scatterers 507 (theinterval of the adjacent light shielding portions 507 b) are equal, andalso these are disposed in parallel, periodicity and symmetry improve inthe entire configuration (particularly, relation between the scatterers503 and the scatterers 507), and the interference properties of lightimproves. Consequently, spectral properties improve.

Further, if the cycle of the opening portions 505 a of the slits 505(the interval of the adjacent opening portions 505 a) and the cycle ofthe scatterers 507 (the interval of the adjacent light shieldingportions 507 b) are equal, and also these are disposed in parallel,periodicity and symmetry improves in the entire configuration(particularly, relation between the slits 505 and the scatterers 507),and the interference properties of light improves. Consequently,spectral properties improve.

Particularly, if the cycle of the scatterers 503 (the interval of theadjacent light shielding portions 503 b), the cycle of the openingportions 505 a of the slits 505 (the interval of the adjacent openingportions 505 a), and the cycle of the scatterers 507 (the interval ofthe adjacent light shielding portions 507 b) are equal, and these aredisposed in parallel, the positional relations of all of the scattererscan be matched, and accordingly, the maximum advantage can be obtainedin periodicity and symmetry. Consequently, the interference propertiesof light improve greatly, and also spectral properties improve greatly.

The present embodiment has features in that the spectral image sensor511 is realized by utilizing such a method for performing spectralseparation using diffraction properties. Such a configuration may be aconfiguration wherein scatterers are embedded in a transparent oxidefilm or nitrogen film which are parent materials. That is to say, it isdesirable that scatterers are embedded in a predetermined transparentmember (parent material) to be integrally formed with a semiconductorsubstrate. In this case, it is desirable to select a member of whichrefractive index is higher than that in a parent member as a scatterer.

Here, there are two reasons regarding why an oxide film and a nitrogenfilm are suitable for a parent member. One reason is in that these filmsare films employed for a common semiconductor process, which areinexpensive and low costs. The other reason is in that these films arelow in a refractive index as compared with silicon, so these films canobtain a great refractive index difference by being combined withsilicon, and consequently, diffraction is effectively generated.

Also, as for a transparent parent material, it is desirable to employ anoxide film or nitrogen film. Here, SiOx is preferably employed as anoxide film, and particularly SiO2 is preferably employed. Also, SiNx ispreferably employed as a nitrogen film. This is because of SiOx,particularly SiO2 which is completely oxidized is chemically stable, andalso its refractive index is the lowest. SiNx is also the same, andSi3N4 is chemically most stable.

Also, it is desirable that scatterers are made up of silicon Si which isexcellent in matching of a process, but scatterers may be made up of theother members, silicon nitride SiN can be employed, for example. At thistime, both of the scatterers 503 and 507 may be made up of the samemember, or may be made up of a different member. SiNx is a materialemployed for a common semiconductor process, and provides an advantageof inexpensive and low costs.

Also, scatterers may be made up of germanium. This refractive index ishigher than that in silicon Si, and an advantage wherein scatteringeffects and diffraction effects are improved can be obtained.

Note that scatterers may be a metal, or compounds thereof as long as adifferent refractive index is employed. That is to say, a transitionmetal, a transition metal silicide, a transition metal nitridingcompound, a transition metal nitriding oxidation compound, a noblemetal, a noble-metal silicide, a refractory metal, a refractory metalsilicide, or the like may be employed. Specifically, Al, Cu, Ni, Cr, W,WSi, WSi2, Ti, TiSi2, TiSi, TiN, TiON, Ta, TaN, NiSi, NiSi2, Hf, HfN,HfSi2, Ag, Au, Pt, CoSi2, RuN, RuOx, RuOxN, or the like can be employed.Of these, inexpensive and low cost scatterers can be obtained byparticularly employing Al, Cu, Ni, or the like.

Note that a sensor configuration utilizing a diffraction grating iseffective even if all of the p-type semiconductors and n-typesemiconductors are reversed. An n-type substrate may be employed for thesake of suppressing noise signals. In this case, such a configurationmay be manufactured following the depth of 2 μm or more from the surfacebeing subjected to p-type formation by subjecting an n substrate tothermal diffusion processing of III family dopant such as Al, B, and soforth. Further, p-type formation may be performed by subjecting thedepth of 0.1 μm or less of the outermost semiconductor surface tothermal diffusion of III family dopant such as Al, B, and so forth forthe sake of suppressing leak current.

Of course, the cycle of the positions of the scatterers 503 (theinterval in the lateral direction from the center point of a lightscatterer to the center point the next same light scatterer) correspondsto the array pitch of the photodiode group 512 (the pixel pitch of thepast configuration), so consequently, a pixel pitch is adjustable bychanging the cycle of the positions of the scatterers 503. In the eventof realizing a high-density image capturing device, this cycle should bedecreased, and inversely, in the event of realizing a low-density imagecapturing device, this cycle should be increased.

For example, even if the cycle of the positions of the scatterers 503 is0.5 through 5 μm, the same advantage can be obtained. The lower limit of0.5 μm is set based on the diffraction limited of visible light. That isto say, in the event of visible light, the diffraction limited is 0.3 μmor more. It is necessary to set a cycle configuration to greater a valuethan at least this value, and accordingly, the lower limit is set to 0.5μm. However, it is not inconvenient to set the low limit to 0.3 μm whichis the original limit.

On the other hand, the upper limit of 5 μm is set based on an orderwhere a diffraction phenomenon markedly occurs. Of course, even if theupper limit is 5 μm or more, a diffraction phenomenon occurs, so theupper limit is not restricted to a particular value. In this point, itis not inconvenient that the term “preferably 5 μm or less” is employedhere.

Also, though it is not necessarily to specify the cycle, the cycle ispreferably a range of 1 through 2 μm, and further preferably 1.5 μl, asdescribed above. Here, the lower limit of 1 μm is set based on the cycleconfiguration of a common diffraction grating of visible light, andfacilitation of manufacturing two or more photoelectric conversiondevices during one cycle. On the other hand, the upper limit of 2 μm isset based on the results of simulation wherein it is confirmed thatmanufacturing up to 2 μm is currently easy.

Also, the scatterers 503 and 507 should be scatterers capable ofobtaining diffraction effects and light harvesting, and either of thescatterers 503 and 507 should have a common thickness of 0.01 μm ormore. Here, the lower limit of 0.01 μm is set based on the minimum valuewherein scattering and diffraction of light can occur. Generally, ifthere are scatterers which are equivalent to around 1/10 of thewavelength of light, scattering and diffraction of light can occur.

Also, the thickness of the scatterer 503 is preferably 0.1 μm or more,further preferably in a range of 0.2±0.05 μm from the relation of therefractive index of SiO2 which is a parent member. Here, the lower limitof 0.1 μm is set based on the thickness wherein scattering anddiffraction are effectively generated. Also, the center value of 0.2 μmis set based on the simulation results wherein particularly excellentspectral properties are exhibited, and range 0.05 μm thereof is set inlight of manufacturing irregularities.

Also, with both of the scatterers 503 and 507, the width of the lateraldirection thereof should be 0.05 μm or more to effectively serve as thescatterers 503 and 507. Here, the lower limit of 0.05 μm is set based onthe minimum value wherein scattering and diffraction of light can occur.Generally, if there are scatterers which are equivalent to around 1/10of the wavelength of light, scattering and diffraction of light canoccur. Note that with this width in the lateral direction, the lowerlimit is preferably set to 0.01 μm in light of matching as to thethickness of the scatterers, but in light of a process, the lower limitdepends on regarding whether or not a 0.05-μm width can be realized atthe latest process. Accordingly, the lower limit is set to 0.05 μm inlight of being capable of scattering and diffraction of light, and theminimum process width.

Also, the width d of scatterers is particularly preferably in a range of0.05 μm≦d≦0.3 μm (0.1−0.05/+0.2 μm). Here, as for generally the centervalue of 0.1 μl, spectral properties are excellent as a result of asimulation, so that this value 0.1 μm is set as the center value. Also,the range of −0.05 μm is restricted considering the lower limit, and onthe other hand, the upper limit is set to +0.2 μm in light ofeasy-to-manufacturing mass productivity process (0.25-μm process).

Also, the width (slit width Ds) of the opening portions 505 a of theslits 505 is 0.1 μm or more, and preferably 0.4 μm or less. Here, thelower limit of 0.1 μm is set based on the minimum value whereindiffraction effectively occurs, and the upper limit of 0.4 μm is setbased on in that the slit width which effectively spectrally separatesvisible light (λ≦780 nm), particularly red light of 640 nm and greenlight of 540 nm is 0.4 μm or less. Note that this does not mean thatspectral separation is not performed at 0.5 μm, but spectral separationis preferably performed at 0.4 μm or less.

Also, particularly in order to improve spectral properties, this slitwidth Ds should be in a range of 0.3±0.1 μm. Here, as for the centervalue of 0.3 μm, spectral properties are excellent as a result of asimulation, so that this value 0.3 μm is set as the center value. Range±0.1 μm thereof is set based on the conditions wherein spectralseparation can be effectively performed using diffraction particularlyin the case of visible light similarly as a result of a simulation. Notethat in the case of infrared spectral separation, the value here resultsin a great difference.

Also, the thickness of the slits 505 is preferably 0.01 μm or more toobtain spectral effects. Here, the lower limit of 0.01 μm is set basedon the minimum value wherein a slit function works. That is to say,function thereof means the thickness wherein light shielding effects, ifany, appears. Also, a particularly effective advantage can be obtainedin a range of 0.125±0.1 μm. Here, as for the center value of 0.125 μm,excellent spectral properties can be obtained as a result of asimulation, so the center value is set to 0.125 μm, the lower limit of−0.1 μm of range±0.1 μm thereof is set based on the determinationwherein the shielding effects of blue light is sufficiently obtained,and the upper limit of +0.1 μm is set for the sake of ease ofmanufacturing.

<<Configuration of Spectral Separation of Infrared Light and VisibleLight>>

FIG. 43 is a diagram describing a demultiplexing image sensorcorresponding to infrared light separation where the diffraction grating501 is disposed at the incident face side of the Si substrate 509(corresponding to the photodiode group 512 of the spectral image sensor510), and illustrates the cross-sectional configuration of the spectralimage sensor 511 for spectrally separating infrared light and visiblelight. The hatching portion illustrates a Si material, a white portionother than that portion illustrates an oxide film SiO2. With thespectral image sensor 511 according to the present embodiment, an oxidefilm SiO2 is formed on the Si substrate 509 as a whole.

Also, the spectral image sensor 511 according to the present embodimentfor spectrally separating into two wavelength components of infraredlight and visible light differs from the spectral image sensor 511 forspectrally separating the inside of a visible light band into multiple(three of blue, green, and red in the previous example) wavelengthcomponents, and has features in that the scatterers 507 are not providedbetween the slits 505 and the photodiode group 512.

In the case of providing no scatterer 507, green light and blue lightare mixed, so when detecting visible light, an arrangement is madewherein a single photoelectric conversion device for detecting visiblelight is provided within the photodiode group 512 in addition to thephotoelectric conversion devices for detecting infrared light, andmonochrome images are obtained without spectrally separating the insideof a visible light band. In order to capture color images, colorseparating filters should be used, for example, subtractive filters(color separating filters) of red, green, and blue for separatingvisible light into each of the three-primary-color light of blue, green,and red are each inserted in front of each of the photoelectricconversion devices (photodetectors) for three-primary-color utilizingthe same method as the past configuration.

The respective long-shaped scatterers 503 are disposed at the incidentface side of the Si substrate 509 (corresponding to the photodiode group512) in the lateral direction (X direction in the drawing) periodicallyin parallel, each of the thicknesses is 0.15 μm, the cycle of thepositions of the respective scatterers 503, i.e., the interval in thelateral direction (X direction in the drawing) from the center point ofthe one scatterer 503 to the center point of the next scatterer 503 is0.2 μm.

With the spectral image sensor 511, at the position of 2.50 μm in thedepth direction (Z direction in the drawing) from surface thereof (theincident side of the incident light L1 of the scatterers 503) and alsoat the position of 1.05 μm from the Si substrate 509, i.e., between thescatterers 503 and the Si substrate 509 (corresponding to the photodiodegroup 512) is provided with the slits 505 having a 0.1-μm thickness and0.80-μm slit width Ds. Consequently, the width of the light shieldingportions 505 b serving as portions for cutting off (shielding in thecase of light) electromagnetic waves (e.g., light), which surrounds theopening portions 505 a, is 1.20 μm.

With the slits 505, the opening portions 505 a each having the slitwidth Ds are disposed in the lateral direction (X direction in thedrawing) periodically in parallel, and further the opening portions 505a each having the slit width Ds are provided at the position (verticalas to the Si substrate 509 and the photodiode group 512 surface) ongenerally the center line CL0 passing through the center point betweenthe scatterers 503, and disposed in parallel with the long-shapedscatterers 503. That is to say, the cycle of the opening portions 505 a(the interval of the adjacent opening portions 505 a) of the slits 505and the cycle (the interval of the adjacent light shielding portions 503b) and phase of the scatterers 503 are equal, and also these aredisposed in parallel.

FIG. 44 is a chart illustrating the relation between the refractiveindex of Si (silicon) and the wavelength dispersion of an extinctioncoefficient which are used for the spectral image sensor 511corresponding to infrared light. Here, let us say that typicalwavelengths of blue, green, and red are 460 nm, 540 nm, and 640 nm,respectively. Also, the refractive index dispersion of an oxide filmSiO2 is extremely small, and accordingly, let us say that the refractiveindex of 1.4, and the extinction coefficient of 0 are applied to anywavelength.

<Simulation of Demultiplexing Technique; Infrared Light and VisibleLight>

FIGS. 45 through 49 are computing simulation diagrams (the results ofoptical field computation using the FDTD method) describing a spectralmethod of infrared light and visible light when the light of eachwavelength components is cast into the spectral image sensor 511 havingthe configuration illustrated in FIG. 43 from the light receiving face(the lower side in the drawing). In FIG. 45 through FIG. 49, thehorizontal dashed line of z=2.5 μm represents the interface (sensorsurface) between the photodiode group 512 and the silicon oxide filmSiO2.

Here, FIG. 45 is a computing simulation result when casting blue light(wavelength of 460 nm), FIG. 46 is when casting green light (wavelengthof 540 nm), and FIG. 47 is when casting red light (wavelength of 640nm), respectively. It can be understood from these drawings that withregard to visible light (any of blue light, green light, and red light),light intensity is strong at the positions of X=−3.0, −1.0, 1.0, and 3.0μm up to Z=2.5 through 3.5 μm (depth of 1.0 μm from the surface of thephotodiode group 512), i.e., near the sensor surface through a somewhatdeep region.

Also, FIG. 48 is a computing simulation result when casting infraredlight (wavelength of 780 nm), and FIG. 49 is when casting infrared light(wavelength of 880 nm), respectively. Here, a wavelength of 780 nm isaround the interface between visible light and infrared light. It can beunderstood from FIG. 49 that with regard to infrared light (wavelengthof 880 nm), light intensity is strong at the positions of X=−2.0, 0, and2.0 μm up to Z=2.5 through 4.5 μm (depth of 2.0 μm from the surface ofthe photodiode group 512), i.e., near the sensor surface through asomewhat deep region.

That is to say, it can be understood that with the relation betweenvisible light (blue light, green light, and red light) and infraredlight included in the incident light L1, visible light and infraredlight are cast into the spectral image sensor 511 illustrated in FIG.43, thereby exhibiting place dependency depending on a wavelength in thewidth direction (X direction in the drawing), and also exhibiting placedependency depending on a wavelength in the depth direction.

It can be understood that in FIG. 49 illustrating a wavelength of 780 nmwhich is around the interface between visible light and infrared light,light intensity is strong at the positions of X=−3.0, −2.0, −1.0, 0,1.0, 2.0, and 3.0 μm up to Z=2.5 through 4.5 μm (depth of 2.0 μm fromthe surface of the photodiode group 512), i.e., near the sensor surfacethrough a somewhat deep region. That is to say, both properties of FIGS.45 through 47 illustrating visible light (blue light, green light, andred light), and FIG. 49 illustrating infrared light (wavelength of 880nm) are exhibited.

<Appropriate Example of Detection Position; Infrared Light And VisibleLight>

FIG. 50 is a diagram describing an appropriate example of a detectionposition in spectral separation of light between visible light andinfrared light based on the above simulation results.

For example, upon configuring a spectral image sensor 511 so as todetect light at areas such as examples illustrated in FIG. 50, a visiblelight band made up of three primary colors of red, green, and blue, andinfrared light (wavelength of 880 nm) can be spectrally separated anddetected.

That is to say, it is desirable to detect visible light (blue light of awavelength of 460 nm, green light of a wavelength of 540 nm, and redlight of a wavelength of 640 nm) at X=−3.0, −1.0, 1.0, and 3.0 μm inZ=2.5 through 3.5 μm (the depth of 1.0 μm from the surface of thephotodiode group 512), and detect infrared light (wavelength of 880 nm)at X=−2.0, 0, 2.0 μm in Z=2.5 through 4.5 μm (the depth of 2.0 μm fromthe surface of the photodiode group 512).

That is to say, it is desirable to set the maximum depth of pn junctionin a range of each Z direction. Specifically, the maximum depth of a pnjunction portion making up a photoelectric conversion device 12W forvisible light is arranged to be set in a range of a depth of 1.0 μm fromthe surface of the photodiode group 512, and the maximum depth of a pnjunction portion making up a photoelectric conversion device 12IR forinfrared light is arranged to be set in a range of a depth of 2.0 μmfrom the surface of the photodiode group 512, thereby improving thedetection efficiency of each wavelength components.

At this time, in order to prevent color mixture among the photodiodegroup 512, the width in the lateral direction (X direction) of each ofthe photodiode group 512 is preferably the interval of 2.0 μm or less inthe lateral direction (X direction in the drawing) of each of thescatterers 503. Also, the width in the lateral direction (X direction)of a pn junction portion making up a photoelectric conversion device bywavelength (visible light and infrared light) to be provided within thephotodiode group 512 is preferably 1.0 μm or less, further preferably0.3 μm or less. These values are values which facilitates massproduction in a semiconductor process.

Here, the lateral width of a pn junction portion is set to 0.5 μm basedon the maximum value from 2.0 μm/2=1.0 μm in the case of considering twospectral separations of infrared light and visible light. Further, 0.3μm is set assuming that this value is mass-producible corresponding tothe current 0.25 μm process. Upon taking into consideration in that thissize is 0.5 μm in the event of spectrally separating only the inside ofa visible light band, it is generally desirable to set this size to 0.5(when a visible light band alone) through 1.0 (when corresponding toinfrared light) μm or less.

<Sensor Configuration Corresponding to Infrared Light DetectionPosition>

FIG. 51 is a cross-sectional view illustrating one configuration exampleof a sensor configuration corresponding to infrared light correspondingto the detection position in FIG. 50. This spectral image sensor 511dopes n-type impurities at the respective detection positions in thewidth direction (X direction in the drawing) and in the depth direction(Z direction in the drawing) corresponding to each of visible light(blue light, green light, and red light) and infrared light for each ofthe photodiode group 512 within the p-type Si substrate 509.

Thus, an n-type Si region 591 for each of visible light and infraredlight is formed, where photoelectric conversion devices (photodiodes)512W and 512IR for each wavelength component are provided. Thephotoelectric conversion devices 512W and 512IR have a configurationwherein visible light, infrared light, visible light, infrared light,and so on are arrayed in the lateral direction (X direction) in thissequence even at the light receiving face side of the Si substrate 509,and also even within the Si substrate 509.

Here, electrons and positive holes are generated by light being absorbedat a depletion layer around the interface between the n-type and p-typeSi semiconductors, and also signal electric charge is pooled by theinterface of a depletion layer moving electrons and positive holes tothe n-type and p-type semiconductors respectively. This signal electriccharge can be detected as an electric signal by being read out from eachof the photoelectric conversion devices 512W and 512IR.

That is to say, with each of the photodiode group 512, detection regionsfor detecting visible light and infrared light independently areprovided at each of the detection positions in the width direction (Xdirection in the drawing) and in the depth direction (Z direction in thedrawing). According to such a configuration, it is unnecessary to use aninfrared light cut filter such as employed for a common image sensor (ora ratio to cut infrared light can be reduced), thereby increasing theamount of light to be cast in per unit area. Accordingly, conversionefficiency between light and an electric signal improves, whereby highsensitivity properties can be also obtained regarding visible light. Inaddition, it is unnecessary to use an infrared light cut filter, therebyrealizing low costs.

This configuration is a configuration wherein the one photodiode group512 detects visible light (blue light, green light, and red light) andinfrared light independently, so one unit of wavelength spectralseparation (wavelength spectral separation units) can be realized by theone photodiode group 512 making up one pixel. That is to say, thisconfiguration is substantially a configuration wherein photoelectricconversion devices are formed by wavelength within the photodiode group512, so it is not necessary to prepare the photodiode group 512corresponding to pixels by wavelength such as for visible light and forinfrared light.

Accordingly, this configuration needs only the one photodiode group 512per one wavelength spectral separation unit, whereby a monochrome imageusing visible light and an infrared image using infrared light can becaptured simultaneously. Thus, the position of the emitting light pointof infrared rays is prepared beforehand to trace this, whereby theposition of the emitting light point of infrared light present within avisible light image (monochrome image) can be detected.

Also, for example, if primary-color filters of R, G, and B are provided,color signals of R, G, and B can be obtained according to a filter colorfrom the photoelectric conversion device 512W, and consequently, avisible light color image can be obtained.

Note that the photoelectric conversion device 512IR serving as aninfrared light image capturing region also serves as a correction pixelas to the visible light image obtained from the photoelectric conversiondevice 512W serving as a visible light image capturing region. Also,upon a pixel including no color separating filter (here, photodiodegroup 512) being provided, this pixel can be taken as a pixel fordetecting the entire wavelength components from visible light toinfrared light, and also can be used as a correction pixel as to thevisible light image obtained from a pixel including another colorseparating filter (here, photodiode group 512).

Note that upon the scatterers 507 being disposed between the slits 505and the light receiving face of the photodiode group 512, green lightand blue light can be effectively spectrally separated. Accordingly, inaddition to the photoelectric conversion device 12IR for detectinginfrared light, instead of the photoelectric conversion device 12W, uponseparate photoelectric conversion devices 512B, 512G, and 512R forreceiving blue light and green light within a visible light band and redlight being provided, a color image using visible light having an exacttone, and an image using infrared light can be captured by one imagesensor simultaneously.

However, wavelength separation properties thereof are not alwayssufficient, the spectral performance of each of red, green, and blue isinferior to the case of the light inside of a visible light band beingspectrally separated into three primary colors of red, green, and blue,and accordingly, color separating filters are preferably employed in thecase of emphasis being put on color reproducibility.

However, upon three primary colors of red, green, and blue beingspectrally separated even if only slightly, an advantage wherein not acomplete monochrome image but a color image can be reproduced can beobtained. Accordingly, even in the case of corresponding to infraredlight, upon the scatterers 507 being disposed between the sensor surfaceand the slits 505, and also the maximum depth of pn junction being setto an appropriate range in each Z direction which can spectrallyseparate and detect the three primary colors of red, green, and blue,not only infrared light and visible light are simply separated, but alsothe light within a visible light band can be separated into the threeprimary color components of red, green, and blue, and accordingly, thedetection efficiency of each color improves. Spectral separation of red,green, and blue within a visible light band and spectral separation ofinfrared light can be realized simultaneously.

<Image Capturing Device; Third Embodiment Utilizing the WavelengthDependency of an Absorption Coefficient in the Depth Direction>

FIGS. 52A and 52B are diagrams for describing a third embodiment of thesolid state image capturing device 314. The solid state image capturingdevice 314 according to the third embodiment is, as with thearrangements described in U.S. Pat. No. 5,965,875, and JapaneseUnexamined Patent Application Publication No. 2004-103964, asingle-plate type utilizing the difference of absorption coefficientsbased on wavelengths in the depth direction of a semiconductor.

Specifically, as the configuration of one pixel worth is illustrated inFIGS. 52A and 52B, the present embodiment employs a solid state imagecapturing device (image sensor) 611 having a configuration wherein avisible light image and an infrared light image are separated andobtained by utilizing the difference of absorption coefficients based onwavelengths in the depth direction of a semiconductor. That is to say,the present embodiment employs an image sensor which can detect thewavelength components which are the original detection target (infraredlight IR components in the present example) while suppressing influenceof the wavelength components other than the original detection target(visible light VL components in the present example) by utilizing thedifference of absorption coefficients depending on a depth and awavelength.

That is to say, in the case of employing an image sensor having aconfiguration utilizing the difference of absorption coefficientsdepending on a wavelength in the depth direction of a semiconductorsubstrate, as for one example, electrons subjected to photoelectricconversion using the visible light VL of less than 780 nm are absorbedat a relatively shallow portion through a depth of around 5 μm (visiblelight detection region 611VL) in the depth direction of the silicon (Si)substrate. Thus, upon employing an arrangement for detecting a signalobtained at the visible light detection region 611VL which is shallowerthan around 5 μm, an electric signal regarding visible light componentscan be obtained.

The remaining light components, i.e., the electrons subjected tophotoelectric conversion using the infrared light IR of which wavelengthis 780 nm or more are absorbed by a region (infrared light detectionregion 611IR) which is deeper than 5 μm. An electric non-contact regionis provided at the interface portion of both detection regions. Thus,upon employing an arrangement for detecting a signal obtained at theinfrared light detection region 611IR which is deeper than 5 μm, anelectric signal of infrared light components can be obtained. That is tosay, the signal components of the visible light VL (e.g., a wavelengthof 780 nm or more) to be subjected to photoelectric conversion at ashallow region of a semiconductor layer are eliminated, and only theinfrared IR (e.g., a wavelength of 780 nm or more) components to besubjected to photoelectric conversion at a deep region of thesemiconductor layer are utilized, thereby obtaining the electric signalof only the incident infrared light IR components.

Thus, with the infrared light detection region 611IR, an infrared lightimage which receives almost no influence of the visible light VL can beobtained. Also, a signal obtained at a relatively shallow portionthrough a region having a depth of around 5 μm is detected, whereby theelectric signal of visible light components can be obtained even at thevisible light detection region 611VL.

Thus, only the photoelectrons made up of incident visible lightcomponents alone are converted into the visible light image capturingsignal SVL to obtain a visible light image which receives almost noinfluence of the infrared light IR, and also only the photoelectronsmade up of incident infrared light components alone are converted intothe infrared light image capturing signal SIR to obtain an infraredlight image which receives almost no influence of the visible light VLsimultaneously and also independently.

Also, as illustrated in FIG. 52B, in order to capture a visible lightcolor image, light receiving face thereof is provided with the colorseparating filter 624 of a predetermined color (e.g., any of 624R, 624G,and 624B) corresponding to each light receiving portion (pixel). Uponutilizing the difference of absorption coefficients depending on awavelength in the depth direction of the semiconductor substrate,wavelength separation of R, G, and B can be performed even at arelatively shallow portion of the visible light detection region 611VL,but actually, separation performance is not so good, and in the event ofemphasis being put on color reproducibility, color separating filtersare preferably employed.

Thus, only the photoelectrons made up of incident visible lightcomponents alone are converted into the visible light image capturingsignal SVL to obtain a visible light color image which receives almostno influence of the infrared light IR, and also only the photoelectronsmade up of incident infrared light components alone are converted intothe infrared light image capturing signal SIR to obtain an infraredlight image which receives almost no influence of the visible light VLsimultaneously and also independently.

Note that an infrared light detection region 611IR of which pixelincluding the color separating filter 424 is relatively deep also servesas a correction pixel as to an visible color image to be obtained fromthe relatively shallow visible light detection region 611Vl. Also, upona pixel including no color separating filter 424 being provided, thispixel can be taken as a pixel for detecting the entire wavelengthcomponents from visible light to infrared light, and also can be used asa correction pixel as to the visible light image to be obtained from apixel including another color separating filter 424.

<Issue Due to Infrared Light Mixture>

As described above, description has been made regarding a configurationexample of the solid state image capturing device 314 having varioustypes of configuration, but any of those includes a problem whereininfrared light components result in being mixed into visible lightcomponents, and being filtered in the detection portion, an infraredsignal results in being added to signal intensity representing a visiblelight image, and consequently, the color reproducibility of a visiblelight color image deteriorates. For example, with the arrangementutilizing a dielectric layered film, advanced features such as highsensitivity, infrared communication function, and so forth can berealized by taking in infrared light and visible light imagessimultaneously, but unless infrared light is completely reflected in thecase of the pixel of an RGB primary-color filter or CyMgYecomplementary-color filter serving as visible light, a part of infraredlight is filtered in the detection portion, and an infrared signal isadded to signal intensity to deteriorate color reproducibility.Description will be made below regarding this point.

FIG. 53 illustrates the spectral image sensor 11 having the sameconfiguration as the spectral image sensor 11 having a dielectricmulti-layered film configuration illustrated in FIG. 36 (thisconfiguration is designed such that the reflection of near-infraredlight improves), but the thickness of the dielectric layer 1 _(—) k ofthe k'th layer is changed to 3.2 μm, and the reflected center wavelengthλ0 of the infrared light IR is not set to 852 nm but is changed to 770nm at a lower side.

FIG. 54 is a diagram illustrating the estimation results by computingthe reflectance spectrum when light is cast in from the verticaldirection using the effective Fresnel-coefficient method with thespectral image sensor 11 having a dielectric multi-layered filmconfiguration illustrated in FIG. 53. Here, the reason why amulti-layered film of SiN and SiO2 is employed is that this is amaterial frequently employed for a common Si family process. Atransmittance spectrum is obtained by subtracting a reflectance R from1, i.e., T=1−R.

Also, FIG. 55 is a diagram illustrating the spectral sensitivity curveof a color filter commonly employed (spectral sensitivity propertiesdiagram). FIG. 56 is a diagram illustrating one example of a spectralsensitivity curve actually obtained. The spectral sensitivity curve,which is actually obtained, illustrated in FIG. 56 is obtained bymultiplying the sensitivity curve illustrated in FIG. 55 by thetransmittance spectrum to be derived from the reflectance spectrumillustrated in FIG. 54. As can be understood from FIG. 56, the infraredlight side of a wavelength of around 700 nm or more has spectralsensitivity of approximate 10%.

Also, FIG. 57 has the same configuration as the spectral image sensor 11having a dielectric multi-layered film configuration (this configurationis designed so as to improve reflection of near-infrared light)illustrated in FIG. 36, as with FIG. 54, and is a diagram illustratingthe results of estimating a reflectance spectrum by computation in theevent that the thickness of the dielectric layer 1 _(—) k of the k'thlayer is set to 3.2 μm, and the reflected center wavelength λ0 of theinfrared light IR is set to 852 nm. Here, a case wherein a centercondition which is a design center, and the case of the reflected centerwavelength λ0 of the infrared light IR as to this center condition areset to −10% and −10%, respectively is illustrated.

FIG. 58 is a diagram illustrating the spectral sensitivity curve of acolor filter employed for the image capturing device ICX456AQmanufactured by Sony Corporation. FIG. 59 is a diagram illustrating aspectral sensitivity curve actually obtained by ICX456AQ. Here, asillustrated in FIG. 53, the thickness of the dielectric layer 1 _(—) kof the k'th layer is set to 3.2 μm, and the basic layer having anine-layer configuration is provided thereupon. As can be understoodfrom comparison between FIGS. 56 and 59, in the event that the reflectedcenter wavelength λ0 of the infrared light IR is high, the infraredlight side of a wavelength of around 700 nm or more further has greatspectral sensitivity.

As illustrated in FIG. 1, most semiconductors have absorptionsensitivity as to infrared light. Accordingly, in the case of theinfrared light side having a certain degree of spectral sensitivity, inaddition to the signal of the light intensity of three primary-color orcomplementary-color visible light, the signal components of infraredlight is added to the pixel signal to be obtained from the detectionportion for receiving visible light, which differs from the actual colorsignal, and consequently, color reproducibility deteriorates.Particularly, such as red in the case of primary-color filters, andmagenta and yellow in the case of complementary-color filters, theamount of infrared light components to be filtered in as to the pixelsignal to be obtained from the detection portion relating to wavelengthcomponents close to infrared light becomes great.

FIGS. 60 and 61 are diagrams describing influence as to colorreproducibility caused by infrared light components being mixed invisible light components. The results obtained by computing colordifference in the case of infrared light being cast into the lightreceiving portion of visible light simultaneously are illustrated, here.All of the results are obtained by computing color difference δEab (moreaccurately, δEa*b*, but “*” is omitted) in a Lab space (accurately,L*a*b*, but “*” is omitted) regarding the 24 colors of the Macbethchart. Note that the Macbeth chart is a chart standardized byGretagMacbeth as a calibration chart of a color measuring apparatus formanagement use.

Here, FIG. 60 illustrates a case wherein the signal intensity of eachcolor all increases +5% by infrared light casting into pixelssimultaneously, and FIG. 61 illustrates a case wherein R increases20.62%, G 10.4%, and B 15.3% respectively. Note that with thesedrawings, the calculated results are plotted on the xy chromaticitydiagrams, and a region satisfying δEab≧5 is illustrated.

The Eab is obtained by computing the case of adding the amount ofincident infrared light, and the case without adding that, such as thefollowing Expression (4).

$\begin{matrix}{\mspace{76mu} \lbrack {{Expression}\mspace{14mu} 4} \rbrack} & \; \\ \begin{matrix}{{\Delta E}_{a^{*}b^{*}} = \{ {( {\Delta \; L^{*}} )^{2} + ( {\Delta \; a^{*}} )^{2} + ( {\Delta \; b^{*}} )^{2}} \}^{\frac{1}{2}}} & \; \\\; & \begin{matrix}{{wherein}\mspace{14mu} {the}\mspace{14mu} {following}} \\{{condition}\mspace{14mu} {is}\mspace{14mu} {{satisfied}.}}\end{matrix} \\{L^{*} = {{25( {100\; \frac{Y}{Y_{0}}} )^{\frac{1}{3}}} - 16}} & {1 \leq Y \leq 100} \\\; & {{{with}\mspace{14mu} D_{55}\mspace{14mu} {light}\mspace{14mu} {source}},} \\{a^{*} = {500\{ {( \frac{X}{X_{0}} )^{\frac{1}{3}} - ( \frac{Y}{Y_{0}} )^{\frac{1}{3}}} \}}} & {X_{0} = 95.68} \\\; & {Y_{0} = 100} \\{b^{*} = {200\{ {( \frac{Y}{Y_{0}} )^{\frac{1}{3}} - ( \frac{Z}{Z_{0}} )^{\frac{1}{3}}} \}}} & {Z_{0} = 90.93}\end{matrix} \} & (4)\end{matrix}$

As can be understood from FIGS. 60 and 61, either of the cases includesa region satisfying δEab≧5 by adding the amount of incident infraredlight to the visible light components. For example, with the case ofFIG. 60, 6 colors of the 24 colors in the Macbeth chart satisfy δEab≧5,and further with case of FIG. 61, 17 colors of the 24 colors satisfyδEab≧5. Generally, upon satisfying δEab≧5, color difference can bedetected by a human sense, and consequently, adding the amount ofincident infrared light to visible light components deteriorates colorreproducibility.

Accordingly, in the case of considering color reproducibility, it can beunderstood that employing an infrared light cut filter is effective.However, employing an infrared light cut filter results in costincreases, and sensitivity deteriorates by the light included in visiblelight being also cut.

As for patent documents describing employing no infrared light cutfilter, with Japanese Unexamined Patent Application Publication No.2001-69519, an arrangement for utilizing an image capturing deviceincluding a filter which transmits no near-infrared region light hasbeen described, but the specific material and configuration of thefilter has not been described. Also, an arrangement wherein the positionof an infrared cut filter is switched described in Japanese UnexaminedPatent Application Publication No. 2000-59798, and an arrangementwherein the sign (positive/negative) of a R-Y signal is detected, andcorrection is performed when the R-Y signal is positive which increasesinfluence of infrared light described in Japanese Unexamined PatentApplication Publication No. 2003-70009 have been proposed. However,these arrangements include a disadvantage wherein the scale of thedevice increases, or the scale of circuits increases, resulting inincrease of costs, a disadvantage wherein correction accuracy is notsufficient, and so forth.

Accordingly, in order to increase the reflectance of infrared light, amethod can be conceived wherein with the spectral image sensor 511utilizing a dielectric layered film employing no infrared light cutfilter, the number of layers of a multi-layered film is increased, orthe refractive index difference of the respective layers of amulti-layered film is increased. However, with the method for increasingthe number of layers, as illustrated in FIG. 8 for example, in the eventof employing the spectral image sensor 11 made up of a combination of aSiO2 layer and a SiN layer, even when employing 6 cycles 11 layers,reflectance thereof is around 0.9 as illustrated in FIG. 9, and thismeans that the amount of incident infrared light is included in visiblelight components, which deteriorates color reproducibility. In order tohave the reflectance approach 1.0, it can be conceived that the numberof layers is further increased, but in this case, the thickness becomes1 μm or more simultaneously. Employing such a thick film configurationaccompanies difficulty on the manufacturing process of a multi-layeredfilm, and consequently, which poses a problem in mass productivity.

Also, in the event of employing a large refractive index difference, andemploying the spectral image sensor 11 made up of a combination of aSiO2 layer and a Si layer such as the fourth modification illustrated inFIGS. 26 and 28 for example, upon employing the basic layer of afive-layer configuration, the reflectance at an infrared region can beincreased up to around “1” as illustrated in FIGS. 27 and 29, but on theother hand, the reflectance at a visible light region is also increased,and the photoelectric conversion efficiency of visible lightdeteriorates, which becomes a factor for deterioration in sensitivity.

Also, there is concern that not a little leakage due to oblique incidentlight exists, influence of the other leakage components is received eachother, the visible light image separated and obtained deteriorates incolor reproducibility for the amount of leakage thereof, and also thevisible light image components for the amount of leakage thereof appearin an infrared light image.

That is to say, in the event of employing a sensor configurationutilizing a dielectric layered film, it is difficult to optimize all ofthe thickness of the device, light receiving sensitivity, colorreproducibility, and so forth, so there is no other choice other than aconfiguration for balancing the entirety, and consequently, colorreproducibility due to leakage of infrared light components remains as aproblem.

Also, in the case of the spectral image sensor 511 utilizing thediffraction grating 501, as can be understood from FIGS. 45 through 49,visible light and infrared light can be separated by utilizing placedependency depending on a wavelength in the width direction (X directionin the drawing), but as can be understood from FIG. 49, separation ofvisible light (blue light, green light, and red light) and infraredlight (wavelength of 880 nm) is incomplete at near the boundary betweenvisible light and infrared light, and consequently, colorreproducibility due to leakage of infrared light components remains as aproblem. Inversely, with regard to an infrared light image, there isinfluence due to leakage of visible light components.

Also, in the case of the solid state image capturing device 611utilizing the difference of absorption coefficients depending on awavelength in the depth direction of the semiconductor, the visiblelight components to be obtained at the visible light detection region611VL are, as can be understood from the past technology describedabove, subjected to a certain degree of absorption when the infraredlight IR passes through, and the infrared light IR is subjected to falsedetection as the visible light VL, thereby receiving influence ofinfrared light components.

Also, in the case of the solid state image capturing device 611 having aconfiguration utilizing the difference of absorption coefficientsdepending on a wavelength in the depth direction of the semiconductor,the wavelength of near the boundary between the infrared light IR andthe infrared components within the visible light VL is subjected to acertain degree of the other absorption each other, so the infrared lightimage components to be obtained at an infrared light image capturingregion sometimes receive influence of a visible light band,particularly, red components.

<<Solution for Infrared Light Mixture>>

In order to solve such a problem, the image capturing apparatus 300according to the present embodiment attempts to solve a colorreproducibility problem due to infrared light mixture at a detectionregion for receiving visible light by providing an infrared lightcorrection processing unit 342 in the image signal processing unit 340.Thus, infrared light which is unnecessary components as to a visiblelight region can be suppressed and eliminated using signal processingwithout providing optical wavelength separating means (a typical exampleis an infrared light cut filter) in front of the image sensor. Even ifthe detection results of a visible light detection portion includingleakage of infrared light, the unnecessary infrared light components canbe suppressed and eliminated by signal processing, so which expands theusage range of the image sensor, when realizing an image capturingapparatus capable of obtaining a visible color image having sufficientcolor reproducibility. Specific description will be made below regardingmethod thereof.

Color Separating Filter Array First Example

FIGS. 62A through 62C are diagrams illustrating a first specific example(hereinafter, referred to as “first specific example”) of the layout ofcolor separating filters which constantly enable a visible light colorimage and an infrared light image to be obtained independently usingcorrection computing. This first specific example has features in that adetection region for eliminating visible light, and receiving anddetecting infrared light alone is provided as a detection region forcorrection as to a visible light color image.

As illustrated in FIG. 62A, color filters having a so-called bayer arraybasic formation wherein the respective color filters are disposed in amosaic shape are employed, first of all, a pixel portion is configuredwherein the repetition units of the color separating filters aredisposed in two pixel by two pixel such that the unit pixels disposed ina tetragonal lattice shape corresponds to the three color filters of red(R), green (G), and blue (B). Also, in order to provide a detectionportion (detection region) for eliminating visible light, and receivingand detecting infrared light alone, one of the two greens (G) issubstituted with a black filter BK. That is to say, filters for threewavelength regions (color components) of primary-color filters R, G, andB for visible light color images, and four types of color filters havingindividual filter properties such as a black filter BK for infraredlight which differs from the components of the primary-color filters R,G, and B are disposed regularly.

For example, a first color pixel for detecting a first color (red; R) isdisposed at an even-line odd-row, a second color pixel for detecting asecond color (green; G) is disposed at an odd-line odd-row, a thirdcolor pixel for detecting a third color (blue; B) is disposed at anodd-line even-row, and a fourth color pixel (here, black correctionpixel) for detecting the infrared light IR is disposed at an even-lineeven-row, G/B pixels or R/BK pixels, which differ for each line, aredisposed in a checkered pattern shape. Such a color array of colorfilters having a bayer array basic formation is repeated for each twocolors of G/B or R/BK regarding any of the line direction and rowdirection.

A visible light color image can be captured by the correspondingdetection portion detecting visible light through the primary-colorfilters R, G, and B, and also an infrared light image can be captured bythe corresponding detection portion detecting infrared light through theblack filter BK independently from a visible light color image and alsosimultaneously with that. Also, the infrared light signal obtained fromthe pixel where the black filter BK is disposed is also utilized as acorrection signal as to the visible light color image to be obtainedfrom the pixels where the primary-color filters of R, G, and B aredisposed.

FIGS. 63 and 64 are diagrams describing a CCD solid state imagecapturing device which is configured so as to have the layout of thecolor separating filters illustrated in FIGS. 62A through 62C, andcapture the infrared light IR alone and the two wavelength components ofthe visible light VL separately as images at the same time. Here, FIG.63 is a sketch (perspective view) illustrating a configuration example.Also, FIG. 64 is a cross-sectional configuration diagram of around thesubstrate surface. Note that here, an application example as to the CCDsolid state image capturing device 101 utilizing a dielectric layeredfilm.

With the configuration of the CCD solid state image capturing device 101illustrated in FIG. 63, only the unit pixel matrix 12 made up of fourpixels is illustrated, but the actual configuration is a configurationwherein the unit pixel matrix 12 made up of four pixels is repeated inthe horizontal direction, and further is repeated in the verticaldirection.

Of four pixels of a cycle array making up the unit pixel matrix 12, theblack filter 14BK in which the dielectric layered film 1 is not formedis provided on one pixel 12IR, which is configured so as to receive theinfrared light IR alone through this black filter 14BK. That is to say,an arrangement is made wherein the black filter 14BK is employed on thepixel 12IR of the infrared light IR as the color filter 14, whereby thevisible light VL can be cut, and the infrared light IR alone can bereceived. The pixel 12IR to be used for correction as to a visible lightcolor image where the black filter 14BK is provided is also referred toas a black correction pixel 12BK.

On the other hand, an arrangement is made wherein the dielectric layeredfilm 1 is formed on the other three pixels 12B, 12G, and 12R, and theprimary-color filters 14R, 14G, and 14B are provided above thereof, thecorresponding three primary colors of blue B, green G, and red R withinthe visible light VL are received through the primary-color filters 14R,14G, and 14B. That is to say, a function for effectively cut infraredlight is realized by forming a dielectric layered film on the detectionportions of pixels where three primary-color filters are provided.

Also, in FIG. 64 in which the cross-sectional configuration diagram ofaround the substrate surface is illustrated, a pixel for receiving thevisible light VL alone is illustrated. The pixel 12IR for receiving theinfrared light IR has a configuration wherein the dielectric layeredfilm 1 and the black filter 14BK are not provided. That is to say,following dielectric layered films being layered by sequentiallylayering SiN layers and SiO2 layers with the configuration illustratedin FIG. 17 using the CVD method such as the manufacturing processdescribed with FIG. 32, only the pixel for receiving the infrared lightIR is removed using lithography technology and the RIE method.Subsequently, a SiO2 layer is layered again and planarized.

We have found that a visible light color image based on three primarycolor components, and an image made up of the infrared light IR alonecan be captured simultaneously by employing an image capturing devicemanufactured with such a configuration. However, there is a concernwherein with a visible light color image, color reproducibility isdeteriorated by leakage of infrared light. Accordingly, correction isperformed as follows.

<Correction Method of First Specific Example>

FIGS. 65A and through 68B are diagrams describing a correction method ofinfrared light components in the first specific example. Here, FIGS. 65Aand 65B are diagrams illustrating a properties example of the colorfilter 14 to be employed for the first specific example. Also, FIGS. 66Athrough 68B are diagrams describing the setting method of a coefficientto be employed for correction calculation.

First, the primary color filters 14R, 14G, and 14B are employed as thecolor filters 14 for visible color image capturing with blue componentsB (e.g., transmittance is generally one at wavelength λ=400 through 500nm, and generally zero at the others), green components G (e.g.,transmittance is generally one at wavelength λ=500 through 600 nm, andgenerally zero at the others), and red components R (e.g., transmittanceis generally one at wavelength λ=600 through 700 nm, and generally zeroat the others), which are the three primary colors of the visible lightVL (wavelength λ=380 through 780 nm), being taken as the center.

Note that transmittance of “generally one” shows an ideal state, and itis desirable that the transmittance in the wavelength region isextremely greater than that in the other wavelength regions. A partthereof may include transmittance other than “1”. Similarly,transmittance of “generally zero” also shows an ideal state, and it isdesirable that the transmittance in the wavelength region is extremelysmaller than that in the other wavelength regions. A part thereof mayinclude transmittance other than “zero”.

Also, any filters can be employed as long as filters pass through thewavelength region components of a predetermined color within a visiblelight VL region (primary color or complementary color) serving aspassage wavelength region components, and whether or not an infraredlight IR region serving as reflected wavelength region components ispassed through, i.e., transmittance as to infrared light IR can beignored. This is because the infrared light IR components are cut by thedielectric layered film 1.

As for one example, color filters having spectral sensitivity propertiessuch as illustrated in FIG. 65A (actually, the same as FIG. 55) areemployed. This is the spectral sensitivity properties of color filterscurrently commonly employed. The sensitivity properties employed hereare arranged wherein a wavelength of around 380 nm through 540 nm istaken as a blue wavelength region, a wavelength of around 420 nm through620 nm is taken as a green wavelength region, and a wavelength of around560 nm through 780 nm is taken as a red wavelength region.

As can be understood from this FIG. 65A, the sensitivity curve of thegreen G has sensitivity even as to long wavelength light which is longerthan 640 nm. This means that color reproducibility deteriorates whenlong wavelength light, which is longer than 640 nm, is casts in. Thesame can be applied to the other colors (R, B), which has sensitivity asto the light of an infrared light region, this means that colorreproducibility deteriorates when infrared light is cast in.

Also, as for the color filter 14 for capturing an infrared light imageto be also used as a correction component as to a visible light colorimage, for an example, as illustrated in FIG. 65B, the black filter 14BKhaving features for principally absorbing the visible light VL, andtransmitting the infrared light IR, i.e., a color filter of whichtransmittance is generally one at the infrared light IR (wavelengthλ≧780 nm), and generally zero at the others is employed. Thus, the pixel12IR is arranged to have sensitivity as to infrared light alone.

Note that transmittance of “generally one” shows an ideal state, and itis desirable that the transmittance at the infrared light IR wavelengthregion is extremely greater than that in the other wavelength regions.Similarly, transmittance of “generally zero” also shows an ideal state,and it is desirable that the transmittance in the wavelength region isextremely smaller than that in the other wavelength regions.

Note that the black filter 14BK mentioned here, essentially, should be afilter for transmitting the long wavelength side which is longer than awavelength of 780 nm which is the boundary of visible light and infraredlight, i.e., a black filter for exhibiting absorbability at principallya visible light wavelength region of 380 nm through 780 nm, and exhibitspermeability at principally an infrared light wavelength region which isa wavelength of 780 nm or more, but as illustrated in this FIG. 65B,should be also a filter for transmitting the long wavelength side whichis longer than a wavelength of 700 nm, i.e., a black filter forexhibiting absorbability at principally a visible light wavelengthregion of 380 nm through 700 nm, and exhibits permeability atprincipally a wavelength region which is a wavelength of 700 nm or more.

Also, the transmission spectrum of the most appropriate black colordiffers depending on the spectral sensitivity curves of theprimary-color filters 14R, 14G, and 14B. For example, in the case of thespectral sensitivity curve illustrated in FIG. 55, the sensitivity curveof the green G has sensitivity as to long wavelength light which islonger than 640 nm, color reproducibility deteriorates when longwavelength light which is longer than 640 nm is cast in, so in order tocorrect this, as for the black color filter 14BK, it is desirable toemploy a filter for transmitting the long wavelength side which islonger than a wavelength of 640 nm, i.e., a black filter for exhibitingabsorbability at principally a visible light wavelength region of 380 nmthrough 640 nm, and exhibits permeability at principally a wavelengthregion which is a wavelength of 640 nm or more.

The black correction pixel 12BK where such a black filter 14BK isdisposed is provided, whereby the pixel 12IR can measure the infraredlight IR alone to be cast into an image capturing device as a signalvalue. Further, the added infrared signal (signal of infraredcomponents) can be cut by subtracting the value obtained by multiplyingthe signal value by a coefficient from each of the three primary-colorsignals or a complementary-color signal. Thus, an image having excellentcolor reproducibility can be obtained even under the circumstances whereinfrared light exists.

Specifically, the respective primary-color signal components SR, SG, andSB representing a visible light color image are corrected using theinfrared light signal components SIR (measurement signal intensity ofinfrared light) representing an infrared light image, thereby obtainingcorrected primary-color signals SR*, SG*, and SB* for reproducing avisible light color image relating to visible light components (firstwavelength region components) from which influence of infrared light(second wavelength region) components, i.e., correct color signalintensity made up of the respective color signal components alone in theoriginal visible light wavelength region.

When performing this correction calculation, as shown in the followingExpression (5-1), the corrected primary-color signals SR*, SG*, and SB*from which influence of leakage components of infrared light (secondwavelength region) is eliminated are obtained by subtracting correctionsignal components obtained by multiplying the infrared light signalcomponents SIR by predetermined coefficients αR, αG, and αB from theprimary-color signal components SR, SG, and SB obtained by adding theleakage signal components of infrared light to the respective colorcomponents in the original visible light wavelength region.

Note that as compared with the case of providing an infrared light cutfilter for reducing the second wavelength region components, in the caseof providing no infrared light cut filter, the three primary-colorsignal components for a visible light color image are increased, so inorder to obtain the equivalent signal level, as shown in the followingExpression (5-2), it is desirable to further subtract nonlinearcorrection signal components obtained by multiplying the infrared lightsignal components SIR by coefficients ∈R, ∈G, and ∈B and theprimary-color signal components SR, SG, and SB. In other words, it isdesirable to add the value obtained by multiplying the infrared lightsignal components SIR by predetermined coefficients ∈R, ∈G, and ∈B tothe color signal components SR, SG, and SB as nonlinear sensitivitycorrection, and then subtract the correction signal components obtainedby multiplying the infrared light signal components SIR by predeterminedcoefficients αR, αG, and αB from the value to which this sensitivitycorrection is added.

Now, when assuming that the coefficients ∈R, ∈G, and ∈B are negative,correction is actually performed by adding nonlinear signal componentsmultiplied by the negative coefficients ∈R, ∈G, and ∈B to the productbetween the infrared light components detected by the second detectionportion and the original pixel signal detected by the first detectionportion.

Thus, respective correction color signals SR**, SG**, and SB** made upof the respective color signal components alone in the original visiblelight wavelength region for reproducing a visible light color imagerelating to visible light components (first wavelength regioncomponents) from which influence of the leakage components of infraredlight (second wavelength region) is eliminated can be obtained withprecision. Note that this correction in accordance with Expression (5-2)need not be performed as to all of the three primary-color signalcomponents, and may be performed particularly as to only the greensignal components of which degree of influence to a luminance signal isstrong.

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 5}  & \; \\ \begin{matrix}{ \begin{matrix}{{SR}^{*} = {{SR} - {\alpha \; R \times {SIR}}}} \\{{SG}^{*} = {{SG} - {\alpha \; G \times {SIR}}}} \\{{SB}^{*} = {{SB} - {\alpha \; B \times {SIR}}}}\end{matrix} \} \mspace{14mu} ( {5\text{-}1} )} \\{ \begin{matrix}{{SR}^{**} = {{{SR}( {1 - {ɛ\; R \times {SIR}}} )} - {\alpha \; R \times {SIR}}}} \\{{SG}^{**} = {{{SG}( {1 - {ɛ\; G \times {SIR}}} )} - {\alpha \; G \times {SIR}}}} \\{{SB}^{**} = {{{SG}( {1 - {ɛ\; G \times {SIR}}} )} - {\alpha \; G \times {SIR}}}}\end{matrix} \} \mspace{14mu} ( {5\text{-}2} )}\end{matrix} \} & (5)\end{matrix}$

Note that in the event of setting the coefficients αR, αG, and αB, it isnecessary to set these so as to sufficiently suppress the filtering-incomponents of infrared light. Here, the filtering-in components ofinfrared light also depend on the intensity of an infrared lightwavelength region included in a light source.

For example, FIG. 66 is a diagram describing influence due to the lightsource of signal intensity to be detected at a visible light detectionregion. Now, let us see about influence of the wavelength spectrums ofsunlight as one example. The intensity of a signal to be detected at avisible light detection region is obtained by further multiplying thespectral sensitivity curve illustrated in FIG. 56 to be actuallyobtained by multiplying the sensitivity curve illustrated in FIG. 55 bythe transmission spectrum to be guided form the reflectance spectrumillustrated in FIG. 54 by the light source wavelength spectrumillustrated in FIG. 66.

FIGS. 67A through 68B are diagrams schematically illustrating thesituation of influence due to light source in the case of an imagecapturing device wherein the spectral image sensor 11 utilizing thedielectric layered film 1 and the color filters 14 are combined.

FIG. 67A illustrates a case wherein the spectral intensity of infraredlight is at the same level as the spectral intensity of visible light,and FIG. 67B illustrates a case wherein the spectral intensity ofinfrared light is lower than the spectral intensity of visible light. Ascan be understood from comparison of FIGS. 67A and 67B, the infraredlight components included in the signal to be obtained from the imagecapturing device where the color filters 14 of predetermined colors aredisposed depends on (is proportional to) the spectral intensity ofinfrared light included in a light source.

Accordingly, in order to appropriately suppress the filtering-incomponents of infrared light without receiving influence of the spectralintensity of infrared light included in a light source, it is desirablenot to constantly subtract a certain amount from the signal to beobtained from the image capturing device where the color filters 14 ofpredetermined colors are disposed, but to have the subtraction amountdepend on (be proportional to) the spectral intensity of infrared lightincluded in a light source. Infrared light components are actuallymeasured, and the value obtained by multiplying the actual measurementinformation by a coefficient is subtracted from the signal to beobtained from the image capturing device where the color filters 14 ofpredetermined colors are disposed, whereby correction can be appliedwith an appropriate amount depending on the intensity of infrared lightunder actual image capturing environment, resulting in excellentcorrection accuracy.

Also, upon determining the most appropriate coefficient under a certainlight source, the same coefficient can be used even if a light sourcecondition changes, so it is unnecessary for a user to adjust the amountof correction depending on an image capturing environment, leading toease of use.

Thus, output of each pixel where four types of color filters aredisposed is subjected to matrix calculation, whereby a visible lightcolor image and a near-infrared light image can be independentlyobtained. That is to say, the four types of color filters havingdifferent filter properties are disposed at the respective pixels of animage capturing device such as a photodiode, and the output of eachpixel where the four types of color filters are disposed is subjected tomatrix computing, whereby three primary-color output for forming avisible color image receiving almost no influence of near-infraredlight, and output for forming a near-infrared light image receivingalmost no influence of visible light can be obtained independently andalso simultaneously.

Particularly, as for a visible light color image, deterioration in colorreproducibility due to filtering of infrared light is corrected withcomputing processing, whereby image capturing having high sensitivity ata dark place and also excellent color reproducibility can be performed.A phenomenon wherein red signal components close to infrared lightbecome great, and a phenomenon wherein luminance at a red portion of apicture image increases can be absorbed, and also improvement of colorreproducibility and sensitivity rise at the time of low illumination canbe balanced without employing a special image capturing device andmechanism. Also, the properties of the black filter 14BK are set inlight of the properties of primary-color filters to be used, whereby theproblem of color reproducibility due to the leakage components at lowerwavelength side than infrared light can be eliminated.

Also, with the same image capturing device, a part of the dielectriclayered film 1 formed integrally on a photodiode is arranged so as notto partially form the dielectric layered film 1, so the problem ofpositioning does not occur, which is different from the case of aseparate optical member having the dielectric layered film 1 where thedielectric layered film 1 is partially not formed is disposed in frontof the image capturing device.

Note that in addition to the pixels of the primary-color filters of R,G, and B, the signal of infrared light is obtained by adding the pixelof a black filter (black correction pixel 12BK), but this blackcorrection pixel 12BK need not always be subjected to a layout formsufficient for capturing an infrared light image in that this iscorrection as to a visible light color image, and the layout of theblack correction pixel 12BK is not restricted to the layout exampleillustrated in FIGS. 62A through 62C wherein the correction pixel 12IR(second detection portion) is disposed as to the pixels 12R, 12G, and12B (first detection portion) for capturing an ordinary image one onone, and accordingly, the black correction pixel 12BK may be disposed atan arbitrary position.

For example, the black correction pixel 12BK may be disposed in placesnear the corner of the image capturing device. Thus, a layout formwherein one correction pixel (second detection portion) is disposed asto multiple pixels (first detection portion) for capturing an ordinaryimage can be realized, and a correction pixel (second detection portion)can be provided while giving almost no influence to the layout form of apixel (first detection portion) for capturing an ordinary image.However, in this case, correction is performed as to the pixel signalsof multiple pixels for capturing an ordinary image using the correctionsignal obtained at one correction pixel, and accordingly, it isdifficult to handle the in-plane irregularities of the correction signal(infrared light signal in the present example).

In order to solve this, a correction pixel (second detection portion)may be inserted in the entire pixel array periodically with a certainnumerical ratio as to the pixels (first detection portions) forcapturing an ordinary image. In the event that there is in-planeirregularities wherein the reflectance of infrared light of a subjectsurface changes depending on a part of subject thereof, correction canbe appropriately performed by inserting a correction pixel in the entirepixel array periodically. The best form for inserting a correction pixel(second detection portion) periodically is a form wherein a correctionpixel (second detection portion) is disposed as to a pixel (firstdetection portion) for capturing an ordinary image one on one.

Thus, the black correction pixel 12BK is combined with the dielectriclayered film 1, whereby an infrared light signal added to a visiblelight pixel signal can be removed effectively, and a visible light colorimage having excellent color reproducibility can be obtained withoutusing a glass infrared light cut filter. Using no glass infrared lightcut filter provides a cost advantage, high transmittance of visiblelight, and high sensitivity. With regard to deterioration of colorreproducibility due to leakage of infrared light in the case ofutilizing the dielectric layered film 1, correction is performed bycomputing processing using infrared light components actually measuredwith the black correction pixel 12BK, and accordingly, image capturingof which sensitivity at a dark place is high, and color reproducibilityis good can be performed, and also a configuration for correction issimple, and correction accuracy is good.

Also, the black correction pixel 12BK is used as a correction pixel, soit is difficult for a signal to be saturated, and a dynamic range isexpanded as compared with a configuration for passing though the entirewavelength components from visible light to infrared light(particularly, near-infrared light) (see a later-described secondspecific example utilizing white correction pixels).

Note that with the above example, the primary-color filters 14R, 14G,and 14B have been employed as the color filters 14 for capturing avisible light color image, but complementary-color filters Cy, Mg, andYe can be also employed. In this case, as illustrated in FIG. 62B, it isdesirable to employ a layout wherein the primary-color filter 14R isreplaced with the complementary-color filter yellow Ye, theprimary-color filter 14G with the complementary-color filter magenta Mg,and the primary-color filter 14B with the complementary-color filtercyan Cy, respectively. Subsequently, one of the complementary-colorfilter magenta Mg wherein the two complementary-color filters magenta Mgexists diagonally is provided with the black filter BK serving as acorrection pixel.

An arrangement is made wherein the dielectric layered film 1 is formedon pixels 12Cy, 12Mg, and 12Ye except for pixels where the black filteris disposed, complementary-color filters 14Cy, 14Mg, and 14Ye areprovided further above thereof, each color of the corresponding colorscyan Cy, magenta Mg, and yellow Ye within the visible light VL isreceived through the complementary-color filters 14Cy, 14Mg, and 14Ye.That is to say, a function for effectively cutting infrared light isrealized by forming a dielectric layered film on the detection portionsof pixels where three primary-color filters are provided.

Also, a combination of filters where the pixel of the black filter BKserving as a correction pixel can be provided are not restricted to acombination of the complementary-color filters of Cy, Mg, and Ye, andthe pixel of the black filter BK serving as a correction pixel can beprovided as to a combination of one of the primary-color filters such asthe green filter G or white filter W and the complementary-colorfilters. For example, as illustrated in FIG. 62C, with a fieldaccumulation frequency interleave method type wherein the twocomplementary-color filters Cy and Mg, and the primary-color filter of Gare combined, one of the two primary-color filters G which are presentwithin four pixels is preferably replaced with the black filter BKserving as a correction pixel.

When performing correction calculation in the case of employing thesecomplementary-color filters, as shown in the following Expression (6),it is desirable to subtract correction signal components obtained bymultiplying the infrared light signal components SIR by predeterminedcoefficients αCy, αMg, αYe, and αG from color signal components SCy,SMg, Sye, and SG obtained by adding the leakage signal components ofinfrared light to the respective color signal components in the originalvisible light wavelength region. Thus, respective correction colorsignals SCy*, SMg*, SYe*, and SG* made up of the respective signalcomponents alone in the original visible light wavelength region forreproducing a visible light color image relating to visible lightcomponents (first wavelength region components) from which influence ofthe leakage components of infrared light (second wavelength region) iseliminated can be obtained.

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 6} \rbrack & \; \\ \begin{matrix}{ \begin{matrix}{{SCy}^{*} = {{SCy} - {\alpha \; {Cy} \times {SIR}}}} \\{{SMg}^{*} = {{SMg} - {\alpha \; {Mg} \times {SIR}}}} \\{{SYe}^{*} = {{SY}^{*} - {\alpha \; {Ye} \times {SIR}}}} \\{{SG}^{*} = {{SG} - {\alpha \; G \times {SIR}}}}\end{matrix} \} \mspace{14mu} ( {6\text{-}1} )} \\{ \begin{matrix}{{SCy}^{**} = {{{SCy}( {1 - {ɛ\; {Cy} \times {SIR}}} )} - {\alpha \; {Cy} \times {SIR}}}} \\{{SMg}^{**} = {{{SMg}( {1 - {ɛ\; {Mg} \times {SIR}}} )} - {\alpha \; {Mg} \times {SIR}}}} \\{{SYe}^{**} = {{{SYe}( {1 - {ɛ\; {Ye} \times {SIR}}} )} - {\alpha \; {Ye} \times {SIR}}}}\end{matrix} \} \mspace{14mu} ( {6\text{-}2} )}\end{matrix} \} & (6)\end{matrix}$

<Application Example to Another Sensor Configuration According to theFirst Specific Example>

With the above first specific example, in the application example to theCCD solid state image capturing device 101 utilizing the dielectriclayered film, one of the two greens (G) has been replaced with the blackfilter BK to provide a detection region for eliminating visible light,and receiving and detecting infrared light alone, but even in the solidstate image capturing device 611 utilizing the difference of absorptioncoefficients depending on a wavelength in the depth direction of thespectral image sensor 511 utilizing the diffraction grating 501 or thesemiconductor, a detection region for eliminating visible light, andreceiving and detecting infrared light alone is provided, an infraredlight image is obtained by using the infrared light signal to beobtained from this detection region, and also this infrared light signalcan be used for correction as to a visible light color image to beobtained from the pixels where the color filters (e.g., primary-colorfilters R, G, and B) for capturing color images.

For example, in the event of applying to the spectral image sensor 511utilizing the diffraction grating 501, the color filter array of thebasic form of a so-called bayer array can be provided. In this case, inthe plane direction, i.e., in a two-dimensional shape, a visible lightimage capturing region for detecting each color is formed at portionswhere the primary-color filters 12R, 12G, and 12B for capturing colorimages, and an infrared light image capturing region for detectinginfrared light image capturing is formed at circumference thereof.

Also, even in the event of applying to the solid state image capturingdevice 611 utilizing the difference of absorption coefficients dependingon a wavelength in the depth direction of the semiconductor, the colorfilter array of the basic form of a so-called bayer array can beprovided. In this case, in the depth direction, i.e., in athree-dimensional shape, a visible light image capturing region fordetecting each color is formed at a relatively shallow region ofportions where the primary-color filters 12R, 12G, and 12B for capturingcolor images, and an infrared light image capturing region for detectinginfrared light image capturing is formed at a region which is furtherdeeper than that region.

In either case, with regard to the spectral sensitivity properties ofthe primary-color filters 12R, 12G, and 12B, it is desirable to use theproperties of which transmittance is generally one at the wavelengthregion of each primary color and an infrared light region, and generallyzero at the others. Using such properties is for reducing decay ofinfrared light components reaching the deep region of the semiconductorso as not to cause deterioration of a signal level to be obtained fromthe infrared light image capturing region to be used as correctioncomponents as to an infrared light image and a visible light colorimage.

Thus, in either case, an infrared light image which receives almost noinfluence of the visible light VL can be obtained independently of avisible light color image. Also, a visible light color image to beobtained at a shallow region of the semiconductor is subjected tocorrection calculation using infrared light image components to beobtained at a deep region of the semiconductor, thereby eliminatinginfluence of infrared light components to be filtered in a visible lightimage, whereby color reproducibility can be improved.

Also, in either case, as can be understood from the features of eachsensor configuration, provides a configuration wherein a detectionregion for obtaining visible light color images and a detection regionfor obtaining infrared light images are separated in the plane directionor depth direction of the semiconductor, and the color filters 14 forobtaining visible light color images are provided, whereby infraredlight components can be also automatically obtained, which is differentfrom the case of utilizing the dielectric layered film, and accordingly,when forming an infrared light image capturing region, it is unnecessaryto dispose a black filter for correction aggressively, the color filterarray of the basic form of a so-called bayer array can be used as it is.Thus, either case does not cause deterioration in the resolution of avisible light color image and infrared light image due to the pixelwhich is originally G being replaced with a black correction pixel, asdescribed later.

Color Separating Filter Array Second Example

FIGS. 69A through 69C are diagrams illustrating a second specificexample (hereinafter, referred to as “second specific example”) of thelayout of color separating filters which constantly enable a visiblelight color image and an infrared light image to be obtainedindependently using correction computing. This second specific examplehas features in that a detection region for receiving and detectinginfrared light and also the entire wavelength components of visiblelight is provided as a detection region for correction as to a visiblelight color image.

As illustrated in FIG. 69A, color filters having a so-called bayer arraybasic form are employed, first of all, a pixel portion is configuredwherein the repetition units of the color separating filters aredisposed in two pixel by two pixel such that the unit pixels disposed ina tetragonal lattice shape corresponds to the three color filters of red(R), green (G), and blue (B). Also, in order to provide a detectionportion (detection region) for receiving and detecting infrared lightand also the entire wavelength regions of visible light, one of the twogreens (G) is substituted with a white filter W. That is to say, filtersfor three wavelength regions (color components) of primary-color filtersR, G, and B for visible light color images, and four types of colorfilters having individual filter properties such as a white filter W forinfrared light which differs from the components of the primary-colorfilters R, G, and B are disposed regularly.

Note that a white correction pixel where the white filter W is disposedis for passing through the entire wavelength components from visiblelight to infrared light (particularly, near-infrared light), andeffectively, a configuration where color filters are not provided can beemployed in this point.

For example, a first color pixel for detecting a first color (red; R) isdisposed at an even-line odd-row, a second color pixel for detecting asecond color (green; G) is disposed at an odd-line odd-row, a thirdcolor pixel for detecting a third color (blue; B) is disposed at anodd-line even-row, and a fourth color pixel (here, white pixel) fordetecting the infrared light IR is disposed at an even-line even-row,G/B pixels or R/W pixels, which differ for each line, are disposed in acheckered pattern shape. Such a color array of color filters having abayer array basic form is repeated for each two colors of G/B or R/Wregarding any of the line direction and row direction.

A visible light color image can be captured by the correspondingdetection portion detecting visible light through the primary-colorfilters R, G, and B, and also an infrared light image or a mixed imageof infrared light and visible light can be captured by the correspondingdetection portion detecting infrared light through the white filter Windependently of a visible light color image and also simultaneouslywith that. For example, employing the pixel data from the pixel 12IR forreceiving the mixed components of the infrared light IR and the visiblelight VL as it is enables the image of the mixed components of theinfrared light IR and the visible light VL to be obtained, wherebysensitivity can be improved. Also, the image of the visible light VL canbe obtained as well as the image of the mixed components of the infraredlight IR and the visible light VL, but taking the difference betweenboth enables the image of the infrared light IR alone to be obtained.Also, the mixed image signal obtained from the pixel where the whitefilter W is disposed is also utilized as a correction signal as to thevisible light color image to be obtained from the pixels where theprimary-color filters of R, G, and B are disposed.

FIG. 70 is a diagram describing a CCD solid state image capturing devicewhich is configured so as to have the layout of the color separatingfilters illustrated in FIGS. 69A through 69C, and capture two wavelengthcomponents of the infrared light IR and the visible light VL separatelyas images at the same time. Here, FIG. 70 is a sketch (perspective view)illustrating a configuration example. Note that here, an applicationexample as to the CCD solid state image capturing device 101 utilizing adielectric layered film. The cross-sectional configuration diagram ofaround the substrate surface is the same as FIG. 64.

With the configuration of the CCD solid state image capturing device 101illustrated in FIG. 70, only the unit pixel matrix 12 made up of fourpixels is illustrated, but the actual configuration is a configurationwherein the unit pixel matrix 12 made up of four pixels is repeated inthe horizontal direction, and further is repeated in the verticaldirection.

Of four pixels of a cycle array making up the unit pixel matrix 12, thecolor filter 14 in which the dielectric layered film 1 is not formed isnot provided on one pixel 12IR, which is configured so as to receive theinfrared light IR without passing through the color filter 14. In thiscase, the pixel 12IR can receive the mixed components of the infraredlight IR and the visible light VL. This pixel 12IR where the colorfilter 14 is not disposed is referred to as a white correction pixel 12Wor an entire area passage pixel.

Thus, the color filter 14 is not inserted regarding the white correctionpixel 12W such that not only the infrared light IR but also the visiblelight contribute to a signal simultaneously at the pixel 12IR where thedielectric layered film 1 was not formed. Thus, substantially, the pixel12IR for infrared light can be made to function not only as a pixel forthe infrared light IR but also as a pixel for both of the infrared lightIR and the visible light VL.

On the other hand, an arrangement is made wherein the dielectric layeredfilm 1 is formed on the other three pixels 12B, 12G, and 12R, and theprimary-color filters 14R, 14G, and 14B are provided above thereof, thecorresponding three primary colors of blue B, green G, and red R withinthe visible light VL are received through the primary-color filters 14R,14G, and 14B. That is to say, a function for effectively cut infraredlight is realized by forming a dielectric layered film on the detectionportions of pixels where three primary-color filters are provided. Asfor the primary-color filters 14R, 14G, and 14B to be employed in thesecond specific example, the same as the first specific exampleillustrated in FIG. 65A can be employed.

We have found that a visible light color image based on three primarycolor components, and an image made up of the infrared light IR alone ora mixed image of the infrared light IR and the visible light VL can becaptured simultaneously by employing an image capturing devicemanufactured with such a configuration. However, there is a concernwherein with a visible light color image, color reproducibility isdeteriorated by leakage of infrared light. Accordingly, correction isperformed as follows.

Correction Method of the Second Specific Example First Example

FIG. 71 is a diagram describing a correction method of infrared lightcomponents in the second specific example. The white correction pixel12W where the color filters 14 are not disposed is provided, whereby asignal value SW which indicates the synthetic components of the infraredlight IR and visible light cast into the image capturing device can bemeasured at the pixel 12IR.

Note that FIG. 71 illustrates assuming that the transmission propertiesof the white filter are equal at a visible light band and at an infraredlight band, but this is not indispensable, or rather the transmissionintensity of an infrared light band may be lower than that of a visiblelight band. It is desirable for the white filter to have propertieswhich can transmit the entire wavelength components of a visible lightband with sufficient intensity, and also can transmit an infrared lightband with sufficient intensity as compared with the transmissionintensity of the primary-color filters R, G, and B (see alater-described pseudo MLT filter).

Further, the added infrared signal (signal of infrared components) canbe cut by subtracting the value obtained by multiplying the infraredlight components to be obtained from the white correction pixel 12W by acoefficient from each of the three primary-color signals or acomplementary-color signal. Thus, an image having excellent colorreproducibility can be obtained even under the circumstances whereinfrared light exists.

However, the signal value SW to be obtained from the white correctionpixel 12W includes not only the infrared light components IR but alsothe visible light components VL, which is different from the case of theblack correction pixel, so it is necessary to estimate the signalintensity SIR of infrared light from which the signal intensity SVL ofthe visible light components VL is eliminated.

Specifically, the respective primary-color signal components SR, SG, andSB representing a visible light color image are corrected using theinfrared light signal components SIR (measurement signal intensity ofinfrared light) representing an infrared light image, thereby obtainingcorrected primary-color signals SR*, SG*, and SB* for reproducing avisible light color image relating to visible light components (firstwavelength region components) from which influence of infrared light(second wavelength region) components is eliminated, i.e., correct colorsignal intensity made up of the respective color signal components alonein the original visible light wavelength region.

When performing the correction calculation of the first example of thecorrection method of the second specific example, as shown in the aboveExpression (5-1), the corrected primary-color signals SR*, SG*, and SB*from which influence of leakage components of infrared light (secondwavelength region) is eliminated are obtained by subtracting correctionsignal components obtained by multiplying the estimated infrared lightsignal components SIR by predetermined coefficients αR, αG, and αB fromthe primary-color signal components SR, SG, and SB obtained by addingthe leakage signal components of infrared light to the respective colorcomponents in the original visible light wavelength region. Of course,the above Expression (5-2) may be applied to the correction calculationof the first example of the correction method of the second specificexample.

Here, as can be understood from FIG. 71, the infrared light signalcomponents SIR represent of the signal value SW to be obtained at thewhite correction pixel 12W, principally the signal intensity of theinfrared light components IR. Thus, the following Expression (7) holdsas to the signal value SVL of the visible light components to beobtained at the white correction pixel 12W. Let us say that the term“infrared light components IR” mentioned here means principally thelight of which wavelength is longer than 640 nm since it is necessary toshield the light at the wavelength side which is longer than around 640nm of the G components from the spectral sensitivity curve of the colorseparating filter illustrated in FIG. 55. Generally, the definition ofinfrared light is invisible light, which is longer wavelength than 780nm, but definition is made as the above, here.

[Expression 7]

SW=SVL+SIR  (7)

On the other hand, the amount of light of the infrared light IR or thevisible light VL has a proportional relation at the subject side and theimage capturing side. That is to say, upon the amount of lightincreasing, the image capturing side also increases in proportionthereto. Accordingly, the relation such as in FIGS. 65A and 65B holds.

For example, it can be conceived that the amount of the visible light VLwhich transmitted the white filter (including the case of disposing nocolor filter 14) is equal to the sum of a value obtained by multiplyingthe amount of visible light which is transmitted at the primary-colorfilters 14R, 14G, and 14B by each coefficient, and accordingly, thesignal intensity SVL of the visible light components VL transmitted atthe white filter is generally equal to the sum of a value obtained bymultiplying the corrected primary-color signal intensity SR*, SG*, andSB* of the visible light components transmitted at the primary-colorfilters 14R, 14G, and 14B by each of the coefficients βR, βG, and βB,which can be represented such as the following Expression (8).

[Expression 8]

SVL=/βR×SR*+βG×SG*+βB×SB*  (8)

Accordingly, the signal intensity SIR of infrared light components totransmit the white filter can be represented such as the followingExpression (9-1). Further, upon substituting Expression (5-1) forExpression (9-1), result thereof can be represented such as Expression(9-2). Upon further summarizing this regarding the infrared lightcomponents IR, result thereof can be represented such as Expression(9-3).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 9} \rbrack & \; \\ \begin{matrix}\begin{matrix}{{SIR} = {{SW} - {SVL}}} \\{= {{SW} - ( {{\beta \; R \times {SR}^{*}} + {\beta \; G \times {SG}^{*}} + {\beta \; B \times {SB}^{*}}} )}}\end{matrix} & ( {9\text{-}1} ) \\{{SIR} = {{SW} - \begin{Bmatrix}( {{\beta \; R \times ( {{SR} - {\alpha \; R \times {SIR}}} )} +}  \\\begin{matrix}{\beta \; G \times ( {{SG} - {\alpha \; G \times {SIR}} +} } \\{\beta \; B \times ( {{SB} - {\alpha \; B \times {SIR}}} )}\end{matrix}\end{Bmatrix}}} & ( {9\text{-}2} ) \\{{SIR} = \frac{{SW} - ( {{\beta \; R \times {SR}} + {\beta \; G \times {SG}} + {\beta \; B \times {SB}}} }{1 - ( {{\alpha \; R \times \beta \; R} + {\alpha \; G \times \beta \; G} + {\alpha \; B \times \beta \; B}} )}} & ( {9\text{-}3} )\end{matrix} \} & (9)\end{matrix}$

Now, when paying attention to the signal components SW to be obtained atthe white correction pixel, and the signal components SR, SG, and SB tobe obtained at the primary-color filter pixels, and assuming thatrespective coefficients are γW, γR, γG, and γB, the coefficients γW, γR,γG, and γB can be represented such as the following Expressions (10-1)through (10-4), and Expression (9-3) can be rewritten to Expression(10-5) by using the coefficients γW, γR, γG, and γB. That is to say, thesignal intensity SIR of the infrared light components alone included inthe signal value SW to be obtained from the white correction pixel 12Wcan be estimated by using the signal components SR, SG, and SB to beobtained at the primary-color filter pixels.

That is to say, the image signal processing unit 340 subjects the signalvalue SW to be obtained from the white correction pixel 12W serving asthe second detection portion to correction using the signal componentsSR, SG, and SB to be obtained at the primary-color filter pixels,whereby the signal intensity SIR of the infrared light components IRalone serving as the second wavelength region components from whichvisible light components (blue components through red components)serving as the first wavelength region components are eliminated.

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 10} \rbrack & \; \\ \begin{matrix}{{\gamma \; W} = \frac{1}{1 - ( {{\alpha \; R \times \beta \; R} + {\alpha \; G \times \beta \; G} + {\alpha \; B \times \beta \; B}} )}} & ( {10\text{-}1} ) \\{{\gamma \; R} = \frac{\beta \; R}{1 - ( {{\alpha \; R \times \beta \; R} + {\alpha \; G \times \beta \; G} + {\alpha \; B \times \beta \; B}} )}} & ( {10\text{-}2} ) \\{{\gamma \; G} = \frac{\beta \; G}{1 - ( {{\alpha \; R \times \beta \; R} + {\alpha \; G \times \beta \; G} + {\alpha \; B \times \beta \; B}} )}} & ( {10\text{-}3} ) \\{{\gamma \; B} = \frac{\beta \; B}{1 - ( {{\alpha \; R \times \beta \; R} + {\alpha \; G \times \beta \; G} + {\alpha \; B \times \beta \; B}} )}} & ( {10\text{-}4} ) \\{{SIR} = {{\gamma \; W \times {SW}} - ( {{\gamma \; R \times {SR}} + {\gamma \; G \times {SG}} + {\gamma \; B \times {SB}}} )}} & ( {10\text{-}5} )\end{matrix} \} & (10)\end{matrix}$

Note that when setting the coefficients γR, γG, and γB, the coefficientsγR, γG, and γrelate to the coefficients βR, βG, and βB, and thecoefficients αR, αG, and αB. The coefficients αR, αG, and αB arepreferably the same as those in the case of the first specific example.The coefficients βR, βG, and βB are set based on the correspondencerelation between the amount of the visible light VL which transmittedthe white filter (including the case of disposing no color filters 14)and the sum of values obtained by multiplying the amount of visiblelight which transmitted the primary-color filters 14R, 14G, and 14B byeach of the coefficients βP, βG, and βB. For example, the respectivecoefficients α, β, and γ are obtained by arithmetic calculation so as toreduce accident error using the Newton's method.

Generally, as illustrated in FIG. 71, in the event that the transmissionproperties in the visible light region of the primary-color filters 14R,14G, and 14B have generally the same shape, the relation among thecoefficients βR, βG, and βB is βR:βG:βB=3:6:1.

Thus, the output of each pixel of four-types of color filters, i.e., theoutput of the pixels where three-types of primary-color filters aredisposed, and the output of the pixel where the white filter 14W isdisposed (practically, color filters are not disposed) are subjected tomatrix computing, whereby each of a visible light color image and anear-infrared light image can be independently obtained. That is to say,the four types of color filters having different wavelength passageproperties (filter properties) are disposed at the respective pixels ofan image capturing device such as a photodiode, and the output of eachpixel where the four types of color filters are disposed is subjected tomatrix computing, whereby three primary-color output for forming avisible color image receiving almost no influence of near-infraredlight, a synthetic image where infrared light and visible light aremixed, or an image of infrared light alone which receives almost noinfluence of visible light using synthetic processing (specifically,difference processing) between mixed components and visible lightcomponents to be obtained from the white correction pixel 12W can beobtained independently and also simultaneously.

For example, employing the pixel data from the pixel 12IR for receivingthe mixed components of the infrared light IR and the visible light VLas it is enables the image of the mixed components of the infrared lightIR and the visible light VL to be obtained, whereby sensitivity can beimproved. Also, the image of the visible light VL can be obtained aswell as the image of the mixed components of the infrared light IR andthe visible light VL, but taking the difference between both enables theimage of the infrared light IR alone to be obtained.

Also, with the same image capturing device, a part of the dielectriclayered film 1 formed integrally on a photodiode is arranged so as notto partially form the dielectric film 1, so the problem of positioningdoes not occur, unlike the case of a separate optical member having thedielectric layered film 1 where the dielectric layered film 1 ispartially not formed being disposed in front of the image capturingdevice.

Particularly, as for a visible light color image, deterioration in colorreproducibility due to filtering of infrared light is corrected withcomputing processing, whereby image capturing having high sensitivity ata dark place and also excellent color reproducibility can be performed.A phenomenon wherein red signal components close to infrared lightbecome great, and a phenomenon wherein luminance at a red portion of apicture image increases can be absorbed, and also improvement of colorreproducibility and sensitivity rise at the time of low illumination canbe balanced at low cost without employing a special image capturingdevice and mechanism.

For example, the signal SW to be obtained from the white correctionpixel 12W includes not only infrared light components but also visiblelight components, so a luminance signal to be obtained based on thepixels where the primary-color filters 14R, 14G, and 14B for capturingvisible light color images are disposed is subjected to correction(actually, addition computing processing) using the signal SVL of thevisible light components, whereby high sensitivity of a visible lightcolor image can be realized independently of color reproducibility.

As illustrated in FIGS. 69A through 69C, an infrared light signal isobtained by adding the pixel of the white filter (white correction pixel12W) in addition to the pixels of the primary-color filters R, G, and B,but it is not always necessary to have a sufficient layout form forcapturing an infrared light image in that the white correction pixel 12Wis correction as to a visible light color image, and accordingly, thelayout of the white correction pixel 12W is not restricted to the layoutexample illustrated in FIGS. 69A through 69C, or rather may be disposedat an arbitrary position. For example, the white correction pixel 12Wmay be disposed in places near the corner of the image capturing device,or may be inserted in the entire pixel array periodically. Particularly,in the event that the reflectance of infrared light of a subject surfacechanges depending on a part of subject thereof, correction can beappropriately performed by inserting the correction pixel in the entirepixel array periodically.

Thus, the white correction pixel 12W is combined with the dielectriclayered film 1, whereby an infrared light signal added to a visiblelight pixel signal can be removed effectively, and a visible light colorimage having excellent color reproducibility can be obtained withoutusing a glass infrared light cut filter. Using no glass infrared lightcut filter provides a cost advantage, high transmittance of visiblelight, and high sensitivity.

With regard to deterioration in color reproducibility due to leakage ofinfrared light in the case of utilizing the dielectric layered film 1,infrared light components are estimated from a signal including theinfrared light components actually measured using the white correctionpixel 12W, and this estimated infrared light components are subjected tocorrection by computing processing, and thus, image capturing can beperformed with high sensitivity at a dark place and excellent colorreproducibility, and also a configuration for correction is simple(however, more complex than the first specific example for the worth ofestimating infrared light components), and further, the estimatedinfrared light components practically become infrared componentsactually measured, thereby providing excellent correction accuracy.

Incidentally, the white correction pixel 12W has sensitivity in a widewavelength region from the visible light VL to the infrared light IR,and accordingly, a signal is readily saturated as compared with thepixels for capturing visible light color images (here, primary-colorpixels where the primary-color filters are disposed), and particularly,this saturation phenomenon sometimes poses a problem in image capturingunder a bright environment. Specifically, it is difficult to obtain anappropriate infrared light image under a bright environment, and alsocorrection as to a visible light color image sometimes becomesunsuitable.

In order to eliminate this saturation problem, for example, with imagecapturing under a bright environment, image capturing may be performedat high speed using exposure control utilizing a shutter function (notrestricted to a mechanical shutter, includes an electronic shutter). Forexample, an arrangement may be made wherein the image capturing deviceis subjected to exposure in a short cycle, a pixel signal is read outfrom image capturing device thereof (specifically, detection portion),and this pixel signal is sent to the pre-processing unit 332 of theimage capturing signal processing unit 330.

In this case, for example, upon performing exposure and signal readoutwith higher a rate than 60 frames per second, an advantage as tosaturation improves. Alternatively, it is desirable to be able to simplyread out a signal in shorter time than 0.01667 seconds (accumulatedtime). In this case, accumulation of electric charge may be read outeffectively in a short period of time by discharging an electric chargesignal to the substrate side using overflow.

Further preferably, upon performing exposure and signal readout withhigher a rate than 240 frames per second, an advantage as to saturationimproves. Alternatively, it is desirable to be able to simply read out asignal in shorter time than 4.16 ms (accumulated time).

Note that pixels to read out electric charge in a short period of time(accumulated time) so as not to saturated thus may be the whitecorrection pixel 12W alone, or may be all of the pixels including theother pixels for capturing visible light color images (here, theprimary-color pixels where the primary-color filters are disposed).

Also, a weak signal at a dark place may be converted into a strongsignal to improve an S/N ratio by integrating the signal read out infurther a short period of time twice or more. For example, according tosuch an arrangement, even if image capturing is performed at a darkplace or bright place, appropriate sensitivity and a high S/N ratio canbe obtained, leading to expanding of the dynamic range. That is to say,performing image capturing at high speed prevents saturation at thewhite correction pixel 12W from occurrence, and also integrating signalsenables a wide dynamic range to be obtained.

Note that with the above example, the primary-color filters 14R, 14G,and 14B have been employed as the color filters 14 for capturing avisible light color image, but complementary-color filters Cy, Mg, andYe can be also employed. In this case, as illustrated in FIG. 67B, it isdesirable to employ a layout wherein the primary-color filter 14R isreplaced with the complementary-color filter yellow Ye, theprimary-color filter 14G with the complementary-color filter magenta Mg,and the primary-color filter 14B with the complementary-color filtercyan Cy, respectively. Subsequently, one of the complementary-colorfilter magenta Mg wherein the two complementary-color filters magenta Mgexists diagonally is provided with the white filter W serving as acorrection pixel.

An arrangement is made wherein the dielectric layered film 1 is formedon pixels 12Cy, 12Mg, and 12 Ye except for pixels where the white filteris disposed, complementary-color filters 14Cy, 14Mg, and 14Ye areprovided further above thereof, each color of the corresponding colorscyan Cy, magenta Mg, and yellow Ye within the visible light VL isreceived through the complementary-color filters 14Cy, 14Mg, and 14Ye.That is to say, a function for effectively cutting infrared light isrealized by forming a dielectric layered film on the detection portionsof pixels where complementary-color filters are provided.

Also, a combination of filters where the pixel of the white filter Wserving as a correction pixel can be provided are not restricted to acombination of the complementary-color filters of Cy, Mg, and Ye, andthe pixel of the white filter W serving as a correction pixel can beprovided as to a combination of one of the primary-color filters, greenfilter G, and the complementary-color filters. For example, asillustrated in FIG. 67C, with a field accumulation frequency interleavemethod type wherein the two complementary-color filters Cy and Mg, andthe primary-color filter of G are combined, one of the two primary-colorfilters G which are present within four pixels is preferably replacedwith the white filter W serving as a correction pixel.

When performing correction computing in the case of employing thesecomplementary-color filters, Expression (6) can be employed, and at thattime, it is desirable to replace the infrared light components SIR suchas the following Expression (11), as can be estimated from Expression(10-5). Note that of Expression (11), the respective components of Cy,Mg, Ye, and G are applied depending on a color filter to be actuallyused, but it is not always necessary to employ all of the colorcomponents, for example, in the case of the color filter layoutillustrated in FIG. 67B, G components are set to zero, or in the case ofthe color filter layout illustrated in FIG. 69C, Mg components are setto zero.

[Expression 11]

SIR=γW×SW−(γCy×SCy+γMg×SMg+γYe×SYe+γG×SG)  (11)

Correction Method of the Second Specific Example Second Example

As described above, description has been made regarding the correctioncomputing of the first example in accordance with the above Expression(5-1) made up of linear terms alone as a correction method in the caseof employing the white correction pixel 12W, but color difference can befurther reduced by performing correction computing to which nonlinearterms are added. This correction method is in common with the aboveExpression (5-2) in that nonlinear terms are considered, but the conceptof syllabuses differs. Description will be made below in detailregarding this point.

When performing the correction computing of the second example of thecorrection method of the second specific example, as shown in thefollowing Expression (12-1), it is desirable to add nonlinear correctionsignal components including two-dimensional signal components (S×IR)obtained by multiplying the product between the components obtained bysubtracting the coefficients ηR, ηG, and ηB from the primary-colorsignal components S (SR, SG, and SB) and the infrared light signalcomponents SIR by the coefficients ωR, ωG, and ωB to the components tobe obtained by the above Expression (5-1). Note that if the coefficientproducts ωR×ηR, ωG×ηG, and ωB×ηB between the respective color componentsare sufficiently small, the components of the coefficient products canbe ignored, and the following Expression (12-2) can also be applied.Note that the infrared light signal components SIR are the same as thatshown in the above Expression (10-5).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 12} \rbrack & \; \\ \begin{matrix}{ \begin{matrix}{{SR}^{*} = {{SR} - {\alpha \; R \times {SIR}} + {\omega \; R \times ( {{SR} - {\eta \; R}} ) \times {SIR}}}} \\{{SG}^{*} = {{SG} - {\alpha \; G \times {SIR}} + {\omega \; G \times ( {{SG} - {\eta \; G}} ) \times {SIR}}}} \\{{SB}^{*} = {{SB} - {\alpha \; B \times {SIR}} + {\omega \; B \times ( {{SB} - {\eta \; B}} ) \times {SIR}}}}\end{matrix} \} \mspace{14mu} ( {12\text{-}1} )} \\{ \begin{matrix}{{SR}^{*} = {{SR} - {\alpha \; R \times {SIR}} + {\omega \; R \times {SR} \times {SIR}}}} \\{{SG}^{*} = {{SG} - {\alpha \; G \times {SIR}} + {\omega \; G \times {SG} \times {SIR}}}} \\{{SB}^{*} = {{SB} - {\alpha \; B \times {SIR}} + {\omega \; B \times {SB} \times {SIR}}}}\end{matrix} \} \mspace{14mu} ( {12\text{-}2} )}\end{matrix} \} & (12)\end{matrix}$

Expression (12-2) is similar to the above Expression (5-2), butExpression (5-2) aims at obtaining the equal signal level by correctingthe amount of the three primary-color signal components for visiblelight color images to be increased in the case of providing no infraredlight cut filter. On the other hand, Expression (12-1) and Expression(12-2) differs from Expression (5-2) in that Expression (12-1) andExpression (12-2) aim at obtaining correct color information withaccuracy so as to reduce color difference.

Expression (12) is a syllabus example in the case of employing theprimary-color filters 14R, 14G, and 14B, but the same can be applied tothe case of employing the complementary-color filters Cy, Mg, and Ye,and further the case of employing a combination of the green filter G orwhite filter W and the complementary-color filters.

When performing correction computing in the case of employing thesecomplementary-color filters, it is desirable to apply the followingExpression instead of the above Expression (12), as one example. Thispoint is the same concept as that in the case of applying the aboveExpression (6) as to the above Expression (5).

$\begin{matrix}{\mspace{76mu} \lbrack {{Expression}\mspace{14mu} 13} \rbrack} & \; \\ \begin{matrix}{ \begin{matrix}\begin{matrix}\begin{matrix}{{SCy}^{*} = {{SCy} - {\alpha \; {Cy} \times {SIR}} + {\omega \; {Cy} \times ( {{SSy} - {\eta \; {Sy}}} ) \times {SIR}}}} \\{{SMg}^{*} = {{SMg} - {\alpha \; {Mg} \times {SIR}} + {\omega \; {Mg} \times ( {{SMg} - {\eta \; {Mg}}} ) \times {SIR}}}}\end{matrix} \\{ {{ {{SYe}^{*} = {{SYe} - {\alpha \; {Ye} \times {SIR}} + {\omega \; {Ye} \times}}} ){SYe}} - {\eta \; {Ye}}} ) \times {SIR}}\end{matrix} \\{{SG}^{*} = {{SG} - {\alpha \; G \times {SIR}} + {\omega \; G \times ( {{SG} - {\eta \; G}} ) \times {SIR}}}}\end{matrix} \} \mspace{14mu} ( {13\text{-}1} )} \\{ \begin{matrix}{{SCy}^{*} = {{SCy} - {\alpha \; {Cy} \times {SIR}} + {\omega \; {Cy} \times {SCy} \times {SIR}}}} \\{{SMg}^{*} = {{SMg} - {\alpha \; {Mg} \times {SIR}} + {\omega \; {Mg} \times {SMg} \times {SIR}}}} \\{{SYe}^{*} = {{SYe} - {\alpha \; {Ye} \times {SIR}} + {\omega \; {Ye} \times {SYe} \times {SIR}}}} \\{{SG}^{*} = {{SG} - {\alpha \; G \times {SIR}} + {\omega \; G \times {SG} \times {SIR}}}}\end{matrix} \} \mspace{14mu} ( {13\text{-}2} )}\end{matrix} \} & (13)\end{matrix}$

Correction Method of the Second Specific Example Third Example

FIGS. 72 and 73 are diagrams describing the correction method of a thirdexample in the second specific example. With Expression (12) andExpression (13), nonlinear correction signal components includingtwo-dimensional signal components obtained by multiplying the product(S×IR) between the infrared light components IR and the original signalcomponents S by a predetermined coefficient are added so as to reducecolor difference, but the correction components of a two-dimensional ormore, high dimensional expression of the difference (S−η) between thecolor signal S an counting η can also be utilized.

For example, as shown in the following Expression (14-1), it isdesirable to add nonlinear correction signal components includingthree-dimensional signal components (Ŝ2×IR) obtained as a whole bymultiplying the product between the square of the components obtained bysubtracting the coefficients ηR, ηG, and ηB from the primary-colorsignal components SR, SG, and SB and the infrared light signalcomponents SIR by the coefficients ωR, ωG, and ωB to the components tobe obtained by the above Expression (5-1).

Also, it is desirable to add nonlinear correction signal components, notrestricted to the multi-dimensional expression of the product betweenthe difference (S−η) and the infrared light components SIR, an absolutevalue expression such as shown in the following Expression (14-2), or acorrection expression for obtaining nonlinear signal components bymultiplying the infrared light components SIR by a conditionalexpression utilizing a one-dimensional expression such as shown in thefollowing Expression (14-3) can be applied.

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 14} \rbrack & \; \\ \begin{matrix}{ \begin{matrix}{{SR}^{*} = {{SR} - {\alpha \; R \times {SIR}} + {\omega \; R \times ( {{SR} - {\eta \; R}} )^{2} \times {SIR}}}} \\{{SG}^{*} = {{SG} - {\alpha \; G \times {SIR}} + {\omega \; G \times ( {{SG} - {\eta \; G}} )^{2} \times {SIR}}}} \\{{SB}^{*} = {{SB} - {\alpha \; B \times {SIR}} + {\omega \; B \times ( {{SB} - {\eta \; B}} )^{2} \times {SIR}}}}\end{matrix} \} \mspace{14mu} ( {14\text{-}1} )} \\{ \begin{matrix}{{SR}^{*} = {{SR} - {\alpha \; R \times {SIR}} + {\omega \; R \times {{{SR} - {\eta \; R}}} \times {SIR}}}} \\{{SG}^{*} = {{SG} - {\alpha \; G \times {SIR}} + {\omega \; G \times {{{SG} - {\eta \; G}}} \times {SIR}}}} \\{{SB}^{*} = {{SB} - {\alpha \; B \times {SIR}} + {\omega \; B \times {{{SB} - {\eta \; B}}} \times {SIR}}}}\end{matrix} \} \mspace{14mu} ( {14\text{-}2} )} \\{ \begin{matrix}\begin{matrix}{{{{When}\mspace{14mu} S} \leq \eta}\mspace{365mu}} \\{{SR}^{*} = {{SR} - {\alpha \; R \times {SIR}} - {\omega \; R \times ( {{SR} - {\eta \; R}} ) \times {SIR}}}} \\{{SG}^{*} = {{SG} - {\alpha \; G \times {SIR}} - {\omega \; G \times ( {{SG} - {\eta \; G}} ) \times {SIR}}}} \\{ {{ {{SB}^{*} = {{SB} - {\alpha \; B \times {SIR}} - {\omega \; B \times}}} ){SB}} - {\eta \; B}} ) \times {SIR}}\end{matrix} \\\begin{matrix}{{{{When}\mspace{14mu} S} > \eta}\mspace{365mu}} \\{{SR}^{*} = {{SR} - {\alpha \; R \times {SIR}} + {\omega \; R \times ( {{SR} - {\eta \; R}} ) \times {SIR}}}} \\{{SG}^{*} = {{SG} - {\alpha \; G \times {SIR}} + {\omega \; G \times ( {{SG} - {\eta \; G}} ) \times {SIR}}}} \\{{SB}^{*} = {{SB} - {\alpha \; B \times {SIR}} + {\omega \; B \times ( {{SB} - {\eta \; B}} ) \times {SIR}}}}\end{matrix}\end{matrix} \} \mspace{14mu} ( {14\text{-}3} )}\end{matrix} \} & (14)\end{matrix}$

For example, FIG. 72 illustrates a graph for plotting blue (B)components, taking the difference between the luminance values of thecolor components following correction computing using the linearexpressions shown in Expression (5-1) and Expression (6-1) and the truevalues (the luminance values of the B components at the time ofemploying an IR cut filter) of color components thereof as a verticalaxis, and further, taking the luminance values following correctioncomputing as a horizontal axis. In the case of this graph, thedifference exhibits convex dependency as to the luminance valuefollowing correction, and as the line segment indicating propertiesthereof, a form similar to a two-dimensional expression such asproperties example 1, or a form wherein dependency changes depending onwhether the luminance value is greater than or smaller than the value ηBsuch as properties example 2 can be obtained.

Accordingly, as illustrated in FIG. 73, upon adding concave dependencyproperties to the above dependency properties, a constant difference canbe obtained. If nothing is done, a state retaining a constant differencecontinues, so color reproducibility poses a problem, but the entiredifference can be eliminated by further taking white balance. Thus, theerror of correction computing can be reduced by subjecting the changingpoint (equivalent to a coefficient η) of difference to computing of anexpression appropriate thereto.

As for an expression indicating concave dependency, the two-dimensionalexpression of the difference (S−η) as shown in the above Expression(14-1), or a common nonlinear format can be applicable thereto, but thisis not always a multi-dimensional expression. For example, as withExpression (14-2) or Expression (14-3), even with an expression whereindependency changes depending on whether the luminance value is greaterthan or smaller than the value ηB, the same advantage can be obtained.

Color Separating Filter Array Third Example

FIGS. 74A through 74C and 75 are diagrams illustrating a third exampleof a specific example (hereinafter, referred to as “third specificexample”) of a layout example of color separating filters whichconstantly enable a visible light color image and an infrared lightimage to be obtained independently using correction computing. Thisthird specific example has features in that a detection region forreceiving and detecting infrared light and also certain wavelengthcomponents within visible light is provided as a detection region forcorrection as to a visible light color image.

As one example, the white correction pixel 12W having the configurationof the second specific example illustrated in FIG. 70 is replaced withthe pixel wherein the green filter 14GIR is disposed. In this case, thepixel 12IR can receive the mixed components with the green of theinfrared light IR and the visible light VL. The pixel 12IR where thegreen filter 14GIR is disposed is referred to as green correction pixel12GIR. With the pixel 12G where the green filter 14G for detectingvisible light is disposed, it is unnecessary to have sensitivity as toan infrared light region, but it is necessary to pay attention in thatthe green correction pixel 12GIR needs to have sensitivity as to aninfrared light region.

In this case, a visible light color image can be captured by thecorresponding detection portion detecting visible light through theprimary-color filters R, G, and B, and also an infrared light image or amixed image of infrared light and green light can be captured by thecorresponding detection portion detecting infrared light through thegreen filter G for correction independently of a visible light colorimage and also simultaneously therewith.

For example, employing the pixel data from the pixel 12IR for receivingthe mixed components of the infrared light IR and the green light as itis enables the image of the mixed components of the infrared light IRand the green light to be obtained, whereby sensitivity can be improved.Also, the image of the visible light VL can be obtained as well as theimage of the mixed components of the infrared light IR and the greenlight, but taking the difference with the green components in thevisible light image VL enables the image of the infrared light IR aloneto be obtained. Also, the mixed image signal obtained from the pixelwhere the green filter G for correction is disposed is also utilized asa correction signal as to the visible light color image to be obtainedfrom the pixels where the primary-color filters of R, G, and B aredisposed.

When performing correction computing processing, Expression (5-1) andExpression (6) can be applied as is, as with the first and secondspecific examples. Also, when estimating the signal SIR of infraredlight components from the signal SGIR, it is desirable to replace thesignal SW in the second specific example with the signal SGIR to beobtained from the green correction pixel 12GIR, and employ the signalSVL of visible light components regarding green components alone, andspecifically, replace Expression (10-5) with such as Expression (15).

[Expression 15]

SIR=γGIR×SGIR−γG×SG  (15)

Note that the signal SGIR to be obtained from the green correction pixel12GIR includes not only infrared light components but also green lightcomponents, so a signal to be obtained from the pixels where theprimary-color filters 14R, 14G, and 14B for capturing visible lightcolor images are disposed is subjected to correction (actually, additioncomputing processing) using the green light components, whereby highsensitivity and high resolution of a visible light color image can berealized.

In this point, the case of employing the white correction pixel 12W inthe second specific example is the same, the signal to be obtained fromthe pixels where the primary-color filters 14R, 14G, and 14B forcapturing visible light color images are disposed is subjected tocorrection (actually, addition computing processing) using the visiblelight components (R, G, and B) to be detected at the white correctionpixel 12W, whereby high sensitivity and high resolution of a visiblelight color image can be realized.

Note that with the third specific example, the primary-color filters14R, 14G, and 14B have been employed as the color filters 14 forcapturing visible light color images, but a combination withcomplementary-color filters Cy, Mg, or Ye, or primary-color filter G canbe also employed.

Also, even with the configurations of the second specific example andthe third specific example, in theory, the white correction pixel 12W orthe green correction pixel 12GIR can be applied to the spectral imagesensor 511 utilizing the diffraction grating 501, and the solid stateimage capturing device 611 utilizing the difference of absorptioncoefficients depending on a wavelength in the depth direction of thesemiconductor by providing the white correction pixel 12W or the greencorrection pixel 12GIR, as with the modification in the first specificexample.

However, as can be understood from the features of each sensorconfiguration, a configuration wherein the detection region forobtaining a visible light color image and the detection region forobtaining an infrared light image are separated in the plane directionor depth direction of the semiconductor, and infrared light componentscan be automatically obtained at the infrared light image detectionregion by providing the color filters 14 for obtaining visible lightcolor images is provided, so it is inadvisable to provide the pixelwhere the white filter or green filter for correction is disposed,aggressively.

Note that with any of the first through third specific examples, thepixel where a color filter for correction is disposed can be used as apixel for detecting infrared light, whereby advanced functions such asoptical communication using infrared light, distance measurement, and soforth can be realized, and also visible light and infrared light can bedetected and captured as an image simultaneously. Thus, with the sameimage sensor, infrared light image information, which is difficult tosee by the eye, corresponding to a visible light image which can be seenby the eye, and a color image of which tone is particularly accurate(color reproducibility is good) can be received simultaneously. Thus,application is expanded as the key device of an information system suchas an infrared camera and so forth.

For example, the position of the emitting light point of infrared raysis prepared beforehand to trace this, whereby the position of theemitting light point of infrared light present within a visible lightimage can be detected. Also, even with no visual light, e.g., even atnight, a clear infrared light image can be obtained by radiatinginfrared light to capture an image, thereby enabling application as animage sensor for crime prevention.

<Other Layout Examples of Color Separating Filters>

FIGS. 76A through 82B are diagrams describing a pixel array in light ofdeterioration in resolution in the case of providing correction pixelsas to a visible light color image in the spectral image sensor 11utilizing the dielectric layered film 1.

With regard to a pixel array, in the event of applying an arrayconfiguration such as in FIGS. 62A through 62C or FIGS. 69A through 69C,a pixel for detecting infrared light (or mixture of infrared light andvisible light) is to be simply added to the visible light pixels of thepast RGB primary-color filters or CyMgYe complementary-color filters (orprimary-color filter G).

For example, the green pixel G for capturing visible light color imagesand the magenta pixel Mg are originally to be replaced with the blackcorrection pixel, white correction pixel, green correction pixel, ormagenta correction pixel, and accordingly, which may cause deteriorationin resolution regarding any of a visible light color image and aninfrared light image. For example, upon one pixel of the past RGB Bayerarray being replaced with an infrared pixel, resolution deteriorates.However, the problem of deterioration in resolution can be eliminated bydevising the layout form of the pixels (e.g., green pixel G) ofwavelength components which greatly contribute to correction pixels andresolution.

At this time, what is important is to dispose the pixels of infraredlight (or the mixture of infrared light and visible light) so as to forma mosaic pattern with a certain grating interval, and also dispose onepixel within the visible light primary-color RGB or complementary-colorCyMgYe pixels so as to form a mosaic pattern with a certain gratinginterval, in the event of employing a color separating filterconfiguration where the respective color filters are disposed in amosaic shape, as with the past.

Here, the term “so as to form a mosaic pattern” means that when payingattention to certain color pixels, these pixels are to be arrayed in agrating shape with a certain grating interval. It is not alwaysindispensable that the color pixels are adjacent to one another. Notethat as for a typical example in the case of employing a layout formwhere color pixels are adjacent to one another, there is a layout formwherein the squares of infrared light pixels and the other color pixelsare alternately arrayed so as to form a checkered pattern.Alternatively, there is a layout form wherein the squares of one pixelwithin the visible light primary-color RGB or complementary-color CyMgYepixels and the other color pixels are alternately arrayed so as to forma checkered pattern.

<Application Example to Primary-Color Filters>

For example, in order to suppress deterioration in resolution of avisible light color image while employing RGB primary-color filters, itis desirable to keep the layout density of the pixels of the visiblelight region G, and replace the remaining R or B pixels of the visiblelight region with black pixels, white pixels, or green pixels forcorrection. For example, as illustrated in FIGS. 76A and 76B, first, thecolor pixels G for detecting the green components of a visible lightregion are disposed at odd-lines and odd-rows, and at even-lines andeven-rows within the unit pixel matrix 12 of two lines by two rows, andblack pixels (FIG. 76A), white pixels (FIG. 76B), or green pixels (notshown) for correction are disposed at even-lines and odd-rows.

Also, with the odd in the row direction of the unit pixel matrix 12, thecolor pixels B for detecting the blue components of a visible lightregion are disposed at odd-lines and even-rows in the unit pixel matrix12 of at the odd-numbered in the line direction, and the color pixels Rfor detecting the infrared components of a visible light region aredisposed at odd-lines and even-rows in the unit pixel matrix 12 at theeven-numbered in the line direction. With the even-numbered in the rowdirection of the unit pixel matrix 12, the layout of the color pixels Band the layout of the color pixels R are inversed. As a whole, therepeated cycle of the color filters 14 is to be completed at the unitpixel matrix 12 of two by two.

In the event of the layout form such as illustrated in FIGS. 76A and76B, the layout form of a checkered pattern wherein the squares of onepixel G within the visible light primary-color RGB pixels and the othercolor pixels are alternately arrayed is employed, and the layout densityof the color pixels G which greatly contribute to the resolution in avisible light color image can be set to the same as that of the bayerarray, and thus, deterioration in resolution of a visible light colorimage is eliminated.

However, the layout density of the color pixels R and the color pixels Bbecome ½ as to the bayer array, so that color resolution deteriorates.However, with human visibility regarding colors, red R and blue Bdeteriorate as compared with green G, so it may be conceived that theabove problem will not become a big problem. On the other hand, withregard to infrared light images utilizing correction pixels, the layoutdensity of the correction pixels become ½ as to the color pixels G fordetecting the green components of a visible light region, so theresolution deteriorates as compared with visible light color images.

For example, the CMOS solid state image capturing device (the pixelcircuit configuration is FIGS. 5A and 5B) having a layer configuration(the cross-sectional configuration diagram corresponding to a pixel forreceiving visible light is FIG. 34) illustrated in FIG. 53 where blackcorrection pixels are disposed with the layout form such as illustratedin FIG. 76A using the black filter 14BK exhibiting transmission spectralproperties such as illustrated in FIG. 77 was manufactured in accordancewith the manufacturing process in FIG. 32, and experimented. As aresult, we have found that the high-resolution color image of the threeprimary-color visible light, and the infrared light image which is lowerresolution than a color image, but relatively high-resolution can becaptured simultaneously.

As can be understood from FIG. 77, the infrared light side exhibitstransmission properties. The infrared light components mixed in thethree primary-color visible light pixels were corrected as with theabove Expression (5-1) using the signal indicating the infrared lightcomponents to be obtained from the black correction pixels, but theproblem of color reproducibility due to infrared light components wasnot caused. As a result, we have confirmed that an image captured evenunder an environment where infrared light exists has excellent colorreproducibility and high sensitivity by being subjected to suchcorrection.

Also, the CMOS solid state image capturing device (the pixel circuitconfiguration is FIGS. 5A and 5B) having a layer configuration (thecross-sectional configuration diagram corresponding to a pixel forreceiving visible light is FIG. 34) illustrated in FIG. 53 where whitecorrection pixels are disposed in the layout form such as illustrated inFIG. 76B was manufactured in accordance with the manufacturing processin FIG. 32, and experimented. As a result, we have found that thehigh-resolution color image of the three primary-color visible light,and the image where infrared light which is lower resolution than acolor image, but relatively high-resolution, and visible light are mixedcan be captured simultaneously, and also the image of infrared lightalone can be captured simultaneously by reducing the intensity of blue,red, and green to be detected at the three primary-color visible lightpixels R, G, and B.

Also, as with the above Expression (10-5), the signal indicatinginfrared light components was extracted from the mixed components of thevisible light components and infrared light components to be obtainedform the white correction pixels (estimation), and the infrared lightcomponents mixed in the three primary-color visible light pixels weresubjected to correction in accordance with the above Expression (5-1)using the extracted signal indicating infrared light components, but theproblem of color reproducibility due to infrared light components wasnot caused. As a result, we have confirmed that an image captured evenunder an environment where infrared light exists has excellent colorreproducibility and high sensitivity by being subjected to suchcorrection.

Also, we have confirmed that the luminance signal to be obtained basedon the three primary-color visible light pixels are subjected tocorrection using the visible light components to be obtained from thewhite correction pixels, whereby high sensitivity of a visible lightcolor image can be realized independently of color reproducibility.

Also, we have confirmed that all of the pixels are exposed in a shortperiod of time so as not to be saturated to read out an electric chargesignal, the signal read out in a short period of time is furtherintegrated twice or more, whereby the signal can be converted into agreat signal, and even if image capturing is performed under a darkenvironment or bright environment, appropriate sensitivity can beobtained, leading to expand a dynamic range.

Also, we have confirmed that even if a configuration wherein the blackcorrection pixels and a multi-layered film are combined such asillustrated in FIG. 76A, or a configuration wherein the white correctionpixels and a multi-layered film are combined such as illustrated in FIG.76B is manufactured not only with a CMOS solid state image capturingdevice but also with a CCD configuration such as illustrated in FIG. 64,the same advantage can be obtained.

Also, in order to suppress deterioration in resolution of an infraredlight image, for example, as illustrated in FIGS. 78A and 78B, it isdesirable to replace the positions of the color pixels G for detectingthe green components of the visible light region illustrated in FIGS.76A and 76B with the positions of the black pixels (FIG. 78A), whitepixels (FIG. 78B), or green pixels (not shown) for correction. In thiscase, the layout form of a checkered pattern wherein the squares of theinfrared light pixels serving as correction pixels and the other colorpixels are alternately arrayed is employed, the layout density of thecorrection pixels can be the same as the case of a bayer array, therebyeliminating deterioration in the resolution of an infrared light image.However, the layout density of the color pixels G which contribute tothe resolution in a visible light color image becomes ½ as to thecorrection pixels, so that the resolution of a visible light color imagedeteriorates as compared with that of an infrared light image. Colorresolution is the same as the case of FIGS. 76A and 76B.

For example, the CCD solid state image capturing device (the pixelcircuit configuration is as in FIGS. 4A and 4B, the cross-sectionalconfiguration diagram corresponding to a pixel for receiving visiblelight is FIG. 64) where black correction pixels are disposed with thelayout form such as illustrated in FIG. 78A using the black filter 14BKexhibiting transmission spectral properties such as illustrated in FIG.77 was manufactured and experimented. As a result, we have found that ahigh-resolution infrared light image, and a visible light image which islower resolution than an infrared light image, but relativelyhigh-resolution can be captured simultaneously.

As can be understood from FIG. 77, the infrared light side exhibitstransmission properties. The infrared light components mixed in thethree primary-color visible light pixels were corrected as with theabove Expression (5-1) using the signal indicating the infrared lightcomponents to be obtained from the black correction pixels, but theproblem of color reproducibility due to infrared light components wasnot caused. As a result, we have confirmed that an image captured evenunder an environment where infrared light exists has excellent colorreproducibility and high sensitivity by being subjected to suchcorrection.

Also, the CCD solid state image capturing device (the pixel circuitconfiguration is as in FIGS. 4A and 4B, the cross-sectionalconfiguration diagram corresponding to a pixel for receiving visiblelight is FIG. 64) where white correction pixels are disposed in thelayout form such as illustrated in FIG. 76B was manufactured andexperimented. As a result, we have found that images in which highresolution infrared light and visible light are mixed can be capturedsimultaneously, and an image of high resolution infrared light alone canbe captured by reducing the intensity of blue, red, and green to bedetected at the three primary-colors R, G, and B, and simultaneously, avisible light color image of which resolution is lower than an infraredlight image, but relatively high-resolution can be captured.

Also, as with the above Expression (10-5), the signal indicatinginfrared light components was extracted from the mixed components of thevisible light components and infrared light components to be obtainedform the white correction pixels (estimation), and the infrared lightcomponents mixed in the three primary-color visible light pixels weresubjected to correction in accordance with the above Expression (5-1)using the extracted signal indicating infrared light components, but theproblem of color reproducibility due to infrared light components wasnot caused. As a result, we have confirmed that an image captured evenunder an environment where infrared light exists has excellent colorreproducibility and high sensitivity by being subjected to suchcorrection.

Also, we have confirmed that the luminance signal to be obtained basedon the three primary-color visible light pixels are subjected tocorrection using the visible light components to be obtained from thewhite correction pixels, whereby high sensitivity of a visible lightcolor image can be realized independently of color reproducibility.

Also, we have confirmed that the electric charge of the white pixelsalone is read out in a short period of time using overflow so as not tobe saturated, the signal read out in a short period of time is furtherintegrated twice or more, whereby the signal can be converted into agreat signal, and even if image capturing is performed under a darkenvironment or bright environment, appropriate sensitivity can beobtained, leading to expansion of dynamic range.

Also, we have confirmed that even if a configuration wherein the blackcorrection pixels and a multi-layered film are combined such asillustrated in FIG. 78A, or a configuration wherein the white correctionpixels and a multi-layered film are combined such as illustrated in FIG.78B is manufactured not only with a CCD solid state image capturingdevice but also with a CMOS configuration, the same advantage can beobtained.

FIGS. 79A through 79C are diagrams describing another pixel array in thecase of providing correction pixels as to a visible light color image.This modified form has features in that multiple colors of color filtersto be disposed at correction pixels are combined. For example, with theexample illustrated in FIGS. 79A through 79C, the first specific exampleand the second specific example are combined, the black filter 14BK andwhite filter 14W each serving as a correction pixel are alternatelydisposed at to the unit pixel matrix 12. Here, FIG. 79A is a combinationof FIGS. 62A through 62C and FIGS. 69A through 69C, FIG. 79B is acombination of FIGS. 76A and 76B, and FIG. 79C is a combination of FIGS.78A and 78B.

According to the layout form of such a combination, for example, thewhite correction pixel 12W can be principally used for high sensitivity,and the black correction pixel 12BK can be used for color correction. Ofcourse, the white correction pixel 12W can be also used for colorcorrection.

<Application Example to Complementary-Color Filters>

Also, in order to suppress deterioration in resolution of a visiblelight color image while employing CyMgYe complementary-color filters, itis desirable to keep the layout density of the pixels of the visiblelight region Mg, and replace the remaining R or B pixels of the visiblelight region with black pixels, white pixels, or green pixels forcorrection. For example, as illustrated in FIGS. 80A and 80B, first, thecolor pixels Mg for detecting the magenta components of a visible lightregion are disposed at odd-lines and odd-rows, and at even-lines andeven-rows within the unit pixel matrix 12 of two lines by two rows, andblack pixels (FIG. 80A), white pixels (FIG. 80B), or magenta pixels (notshown) for correction are disposed at even-lines and odd-rows. Note thatone of the magentas Mg can be also replaced with the green G.

In this case, the layout form of a checkered pattern wherein the squaresof one pixel Mg within the visible light complementary-color CyMgYepixels and the other color pixels are alternately arrayed is employed,and the layout density of the color pixels Mg which greatly contributeto the resolution in a visible light color image can be set to the sameas that of the bayer array, and thus, deterioration in resolution of avisible light color image is eliminated.

Note that the layout density of the color pixels Cy and the color pixelsYe is ½ as to the array of the color pixels Mg, so color resolutiondeteriorates, but human visibility regarding colors is low, andaccordingly, it may be conceived that the above problem will not becomea big problem. On the other hand, with regard to infrared light imagesutilizing correction pixels, the layout density of the correction pixels(infrared light pixels) become ½ as to the color pixels Mg for detectingthe magenta components of a visible light region, so the resolutiondeteriorates as compared with visible light color images.

Also, in order to suppress deterioration in resolution of an infraredlight image, for example, as illustrated in FIGS. 81A and 81B, it isdesirable to replace the positions of the color pixels Mg for detectingthe magenta components of the visible light region with the positions ofthe black pixels (FIG. 81A), white pixels (FIG. 81A), or magenta pixels(not shown) for correction. In this case, the layout form of a checkeredpattern wherein the squares of the infrared light pixels serving ascorrection pixels and the other color pixels are alternately arrayed isemployed, the layout density of the correction pixels can be the same asthe case of a bayer array, thereby eliminating deterioration in theresolution of an infrared light image. However, the layout density ofthe color pixels Mg which contribute to the resolution in a visiblelight color image becomes ½ as to the correction pixels, so that theresolution of a visible light color image deteriorates as compared withthat of an infrared light image. Color resolution is the same as thecase of FIGS. 80A and 80B.

Note that with the above layout form example for suppressingdeterioration in resolution, the pixels of the green G or the magenta Mgare disposed so as to form a mosaic pattern (checkered pattern servingas a standard example) with high density as much as possible, but evenif the pixels of the other colors (R, B, or Cy, Ye) are disposed so asto form a checkered pattern, almost the same advantage can be obtained.Of course, in order to improve resolution and color resolution, it isdesirable to dispose the color component filter of which visibility ishigh so as to form a mosaic pattern with high density as much aspossible.

<Application Example to Oblique Layout>

Note that with the above example, description has been made regarding aninstance for disposing color filters in a tetragonal grating shape, butwhich can be arrayed in an oblique grating shape. For example, thelayout form illustrated in FIG. 82A is a pixel array in a state in whichthe layout form illustrated in FIG. 76B is rotated clockwise bygenerally 45 degrees. Also, the layout form illustrated in FIG. 82A is apixel array in a state in which the layout form illustrated in FIG. 78Bis rotated clockwise by generally 45 degrees. Thus, upon color filtersbeing arrayed in an oblique grating shape, each pixel density in thevertical direction and in the horizontal direction is to increase,whereby the resolution in those directions can be further improved.

<<Experimental Example; Black Correction Pixel>>

FIGS. 83 through 94 are diagrams describing the experimental example ofan arrangement for correcting the color reproducibility of a visiblelight color image using the black correction pixels.

First, FIG. 83 is a diagram illustrating the overview of the monochromecamera employed for an experiment in the case of applying blackcorrection pixels. With this experimental example, an experiment wasperformed with the monochrome camera XCL-X700 manufactured by SonyCorporation serving as the base by adding color filters. The fundamentalcapability of the monochrome camera XCL-X700 (hereinafter, also referredto as experimental camera) is, for example, ½-inch type, all pixelsreadout (progressive), the effective number of pixels is 1034×779, thenumber of image capturing pixels is 1024×768, and the size of pixels is6.25 μm.

FIG. 84 is a spectral sensitivity properties diagram of the experimentalcamera and color filters. The entire wavelength pixels in the drawingare the properties itself of the monochrome experimental camera. Also,the respective pixels of R, G, and B are in the case of the respectivecolor filters of R, G, and B are disposed in the experimental camera,and exhibit the results of adding correction of weighting.

FIG. 85 is a diagram of the respective transmission spectrums of a blackfilter employed for G color, an infrared light cut filter, andcorrection pixels. The ideal transmission properties of the infraredlight cut filter is “1” at a visible light region (wavelength of lessthan 700 nm), and “0” at an infrared light region (wavelength of 700 nmor more), but as illustrated in the drawing, there is somewhat loss at avisible light region, and also there is somewhat transmission at aninfrared light region.

Accordingly, with the transmission spectrum of the G color,transmittance somewhat differs (needless to say, transmittance is highin the case of no infrared light cut filter) depending on whether or notthe infrared light cut filter exits. Also, in the case of no infraredlight cut filter, there is somewhat transmission at an infrared lightregion, i.e., there are the leakage components (IR leakage light) ofinfrared light. The leakage components of infrared light can be obtainedseparately from visible light components by using the black filter.

FIG. 86 is a diagram illustrating correspondence of color chip numbers(one cycle; 24 colors) in a Macbeth chart employed as the indices ofcolorimetry. FIGS. 87A through 87C are diagrams illustrating the image(raw data image) based on unfiltered image data obtained by capturing aMacbeth chart using the experimental camera and the green filter G. Asfor image capturing conditions, let us say that a 20-W incandescent lampand a fluorescent light are used as a light source, diaphragm is f2.8corresponding to the lens F2.8 of the experimental camera, and shutterspeed is 1/2.8 second.

Here, FIG. 87A is the green image G in the case of employing no infraredlight cut filter (excluding IR cut), FIG. 87B is the green image G inthe case of employing an infrared light cut filter (including IR cut),and FIG. 87C is difference image thereof, i.e., the image of theinfrared light leakage components of the green filter G. FIG. 88 is adiagram representing a signal level (actual value) for each color chipnumber of the Macbeth chart which is the image capturing resultillustrated in FIGS. 87A through 87C.

As can be understood from comparison of the respective drawings in FIGS.87A through 87C or 88, the signal level of the G color filter pixeloutput which captured the respective color chips of the Macbeth chartdiffers depending on whether or not an infrared light cut filter exists.

FIGS. 89A and 89B are diagrams illustrating the image (raw data image)based on unfiltered image data obtained by capturing the Macbeth chartusing the experimental camera and the black filter BK serving as acorrection pixel. The image capturing conditions are the same as thosein FIGS. 87A through 87C. Here, FIG. 89A is the image of the blackfilter BK (black filter image Br), and FIG. 89B is the image of infraredlight leakage components of the green filter G as a comparative example(the same as FIG. 88C).

FIGS. 90A and 90B are diagrams illustrating a black correction imageBrcorr obtained by multiplying the black filter image Br by apredetermined coefficient uG. The image capturing conditions are thesame as those in FIGS. 87A through 87C. Here, FIG. 90A is the image ofthe black filter image Brcorr in the case of the coefficient αG=0.18,and FIG. 90B is the image of infrared light leakage components of thegreen filter G as a comparative example (the same as FIG. 87C).

As can be understood when comparing both images, any of the signallevels of the respective color chips of the Macbeth chart is generallyin the same state. That is to say, the leakage components of infraredlight which appear on the output of the green filter G can be obtainedseparately from visible light components by using the black filter BK.Accordingly, a green corrected image G* can be obtained by taking thedifference between the image of the green image G (including IR cut)illustrated in FIG. 87B and the black correction image Brcorr, and theinfrared light leakage components can be removed from the green imagewithout employing an infrared light cut filter.

For example, FIGS. 91A through 91C and 92 illustrate one example of thecorrection effects as to the G color image employing the blackcorrection image. Here, FIG. 91A is the green image G (excluding IR cut)(the same as FIG. 87A), FIG. 91B is the green image G (including IR cut)(the same as FIG. 87B), and FIG. 91C is the difference image between thegreen image G (excluding IR cut) and the black correction image Brcorr,i.e., the green corrected image G*. Also, FIG. 92 is a diagramrepresenting a signal level (actual value) for each color chip number ofthe Macbeth chart which is the difference result (green corrected image)illustrated in FIG. 91C.

As can be understood from comparison between FIG. 91A and FIG. 91C, orFIG. 92, any of the signal levels of the respective color chips of theMacbeth chart is generally in the same state. That is to say, theinfrared light leakage components can be almost removed from the greenimage without employing an infrared light cut filter. Almost the samesignal as in the case of employing an infrared light cut filter can beobtained without employing an infrared light cut filter, andaccordingly, almost the same visible light color image having sufficientcolor reproducibility as in the case of employing an infrared light cutfilter can be obtained.

However, as can be understood from FIG. 92, there is somewhat differencewith the case of including an infrared light cut filter. It can beconceived as illustrated in FIG. 85 that this difference is caused bythe transmission spectrum at the visible light region of the G colorhaving a difference depending on whether or not an infrared light cutfilter exists. Accordingly, in order to correct this influence, it isconceived that the green image G (excluding IR cut) is preferablysubjected to sensitivity correction beforehand without simply taking thedifference between the green image G (excluding IR cut) and the blackcorrection image Brcorr. When performing this sensitivity correction, itis desirable to employ the pixel signal of the black filter BK.

That is to say, when performing correction computing in this case, asshown in Expression (5-1), it is desirable to perform correction withhigh precision not only by subtracting the correction signal componentsobtained by multiplying the infrared light signal components SIR bypredetermined coefficients αR, αG, and αB from the color signalcomponents SCy, SMg, SYe, and SG obtained by adding the leakage signalcomponents of infrared light to the respective signal components in theoriginal visible light wavelength region, but also as shown inExpression (5-2), by subjecting the color signal components SCy, SMg,SYe, and SG to sensitivity correction beforehand using the valueobtained by multiplying the infrared light signal components SIR bypredetermined coefficients ∈R, ∈G, and ∈B, and subtracting thecorrection signal components obtained by multiplying the infrared lightsignal components SIR by predetermined coefficients αR, αG, and αB fromthe color signal components SCy, SMg, SYe, and SG subjected to thissensitivity correction.

For example, FIGS. 93A through 94 illustrate one example of correctionresults with high precision thereof. Here, FIG. 93A is the green image G(including IR cut) (the same as FIG. 86B), and FIG. 93B is the greencorrected image G** in the case of performing correction with highprecision assuming that coefficient αG is equal to 0.11, and ∈G isgenerally equal to 0.0012 in accordance with Expression (5-2). Also,FIG. 94 is a diagram representing a signal level (actual value) for eachcolor chip number of the Macbeth chart which is the difference result(green corrected image G**) illustrated in FIG. 93B.

As can be understood from comparison between FIG. 93A and FIG. 93B, orFIG. 94, any of the signal levels of the respective color charts of theMacbeth chart is generally in the same state. That is to say, theinfrared light leakage components can be almost removed from the greenimage without employing an infrared light cut filter. The infrared lightleakage components can be almost completely removed from the green imagewithout employing an infrared light cut filter. The same signal as inthe case of employing an infrared light cut filter can be obtainedwithout employing an infrared light cut filter, and accordingly, thesame visible light color image having sufficient color reproducibilityas in the case of employing an infrared light cut filter can beobtained.

For example, the signal SW to be obtained from the white correctionpixel 12W includes not only infrared light components but also visiblelight components, so a luminance signal to be obtained based on thepixels where the primary-color filters 14R, 14G, and 14B for capturingvisible light color images are disposed is subjected to correction(actually, addition computing processing) using the signal SVL of thevisible light components, whereby high sensitivity of a visible lightcolor image can be realized independently of color reproducibility.

<<Experimental Example; White Correction Pixel>>

FIGS. 95 through 102 are diagrams describing the experimental example ofan arrangement for correcting the color reproducibility of a visiblelight color image using the white correction pixels.

FIG. 95 is a diagram illustrating environmental conditions at the timeof an experiment in the case of applying white correction pixels. Withthis experimental example, we have performed an experiment with themonochrome camera XCX495M manufactured by Sony Corporation (hereinafter,also referred to as experimental camera) being taken as the base,providing the primary-color pixels 12R, 12G, and 12B by adding theprimary-color filters R, G, and B, and also configuring the whitecorrection pixel 12W by providing a pseudo MLT filter. Also, in order toverify the result by providing the white correction pixel 12W, we havemade an arrangement wherein it can be switched to include or exclude anIR cut filter (C5000) instead of the pseudo MLT filter.

Here, “pseudo MLT filter” is an IR cut filter which is generally equalto the transmission properties of Si3N4/SiO2 multi-layered film (5cycles), and the thickness of this filter is 0.4 mm which is thin ascompared with the thickness of 1.6 mm of the filter C5000 manufacturedby Daishinku Corp.

Note that as for a pixel array in the case of providing the whitecorrection pixel 12W, a pixel array such as the above FIGS. 69A, 76B,78B, and so forth can be employed.

Also, three types of halogen lamp of color temperatures 2600 K, 2800 K,and 3000 K, and three types of fluorescent light ECW of colortemperatures 3000 through 7000 K (specifically, each of fluorescentlights of daylight color, daytime white, lamp-bulb color) were taken asa light source condition, the Macbeth chart to be used as indices ofcolorimetry and a resolution chart for resolution evaluation weresubjected to image capturing.

FIG. 96 is a diagram illustrating the transmission properties of anordinary IR cut filter C5000 and a pseudo MLT filter. As can beunderstood from the drawing, all wavelength components of a visiblelight band can be transmitted with sufficient intensity by employing apseudo MLT filter instead of the IR cut filter C5000, and also with aninfrared light band, properties for transmitting the components withsufficient intensity as compared with the transmission intensity of theprimary-color filters R, G, and B can be obtained.

FIG. 97 is a spectral sensitivity properties diagram of color filters inthe case of applying the experimental camera and the pseudo MLT filter.As can be understood from the drawing, in the event of employing thepseudo MLT filter, the leakage components (IR leakage light) of infraredlight (IR) exist in the primary-color filter components of R, G, and Bat an infrared light region (wavelength of 700 nm or more).

FIG. 98 is a flowchart illustrating the overall procedures. Thedifference between the case of employing the IR cut filter C5000 and thecase of employing the pseudo MLT filter is in whether or not theinfrared light correction processing unit 342 includes color correctioncomputing processing (S104) to which the above Expression (5) andExpression (12) are applied.

That is to say, first, the image capturing signal processing unit 330obtains a raw image capturing signal (raw output) from the experimentalcamera (S100). The pre-processing unit 332 subjects the raw imagecapturing signal output from the experimental camera, i.e., the sensoroutput signal (visible light image capturing signal SVL (in detail, therespective color components SR, SG, and SB of R, G, and B) and infraredlight image capturing signal SIR) to pre-processing such as black leveladjustment, gain adjustment, gamma correction, or the like (S102).Subsequently, in the event of employing the pseudo MLT filter, theinfrared light correction processing unit 342 of the image signalprocessing unit 340 executes color correction computing processing towhich Expression (5) and Expression (12) are applied (S104).

Further, the image signal processing unit 340 executes standardizingprocessing for white balance (S106) and standardizing processing of abrightness signal (S108), thereby obtaining bitmap data (S110).Subsequently, the image signal processing unit 340 obtains color datafor evaluation (S120).

When obtaining color data for evaluation (S120), first, R, G, and Bsignals (here, sRGB signals) are subjected to linearization processing(S122), and converted into a XYZ color-coordinate-system signal of threecolor coordinate system based on color-matching functions x(λ), y(λ),and z(λ) adopted by the CIE (Commission Internationale d'Eclairage) in1931 (S124). Further, the above signals are converted into the colorsignals of a Lab coordinate system which is one of uniform color spacedetermined by the CIE in 1976 (S126).

Thus, upon the respective Lab signals in the case of employing the IRcut filter C5000 and in the case of employing the pseudo MLT filterbeing obtained, the respective color differences δEab are obtained inaccordance with the above Expression (4), and the respective colordifferences are compared (S130).

The respective coefficients α, β, and γ in the case of applyingExpression (5) and Expression (12) are obtained using arithmeticalcomputation so as to reduce errors using the Newton method, but here, asone example, let us say that α and γ are such as the followingExpression (16-1). Also, let us say that the respective coefficients ω,and η in the case of applying Expression (12) are such as the followingExpression (16-2).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 16} \rbrack & \; \\ \begin{matrix}{ \begin{matrix}{{\alpha \; R} = 2.1843} \\{{\alpha \; G} = 1.0279} \\{{\alpha \; B} = 0.6065} \\{{\gamma \; R} = 0.0901} \\{{\gamma \; G} = 0.3485} \\{{\gamma \; B} = 0.0936} \\{{\gamma \; W} = 0.169}\end{matrix} \} \mspace{14mu} ( {16\text{-}1} )} \\{ \begin{matrix}{{\omega \; R} = 0} \\{{\omega \; G} = 0} \\{{\omega \; B} = 0.0087} \\{{\eta \; R} = 12} \\{{\eta \; G} = 12} \\{{\eta \; B} = 12}\end{matrix} \} \mspace{14mu} ( {16\text{-}2} )}\end{matrix} \} & (16)\end{matrix}$

In Expression (16-2), the coefficients are obtained so as to reduce acolor difference to the minimum based on the result of measuring theprimary-color filters of the experimental camera, the values of ωR andωG are “0”, but the coefficients are inherent in a device, so the valuediffers depending on a device, and accordingly, in the event of anotherdevice, these values sometimes may be values other than “0”.

FIG. 99 is a diagram illustrating the results of capturing 24 colors ofa Macbeth chart under an environment of a halogen light source (colortemperature of 3000 K), and obtaining the color difference before andafter correction by computing. This FIG. 99 is a diagram which isconverted into a graph so as to compare the color difference whenperforming no correction, the color difference by correction computingof linear terms alone to which Expression (5) is applied, and the colordifference by correction computing including nonlinear terms to whichExpression (12) is applied. Here, the horizontal axis represents thenumber of each color of the Macbeth chart.

As can be understood from FIG. 99, the average color differencefollowing correction is 3.95 which is the limit in the event of applyingExpression (5) as to the average color difference prior to correction of7.36. However, it can be understood that the color difference can befurther reduced to 3.01 by further improving Expression (5), applyingExpression (12), and being subjected to correction using nonlinearterms, and color reproducibility can be improved.

FIG. 100 is a chart summarizing the measurement results of colordifferences regarding two-types of halogen lamp of color temperatures2600 K and 2800 K, and three-types of fluorescent light of daylightcolor, daytime white, and lamp-bulb color in addition to a halogen lightsource (color temperature of 3000 K). Here, the case of no correctionand the case of correction applying Expression (12) are illustrated.

As can be understood from FIG. 100, even under any light source, the IRleakage light is subjected to correction applying Expression (12) andusing nonlinear terms, whereby the average color difference can besufficiently reduced as compared with prior to correction. It can beunderstood that even in the event of a halogen light source (colortemperature of 2800 K), δEab=4.064<5 can be hold, sufficient colorreproducibility can be obtained by correction to which Expression (12)is applied even if the incident amount of infrared light are mixed invisible light components. Note that in the event of a halogen lightsource (color temperature of 2600 K), δEab=6.14>5, and accordingly,color reproducibility is still at stake.

<Regarding Noise>

Note that when performing correction computing by applying Expression(5), Expression (12), or the like, there is concern regarding noise(S/N) deterioration accompanied with this correction computing. However,according to the experiment, we have found that there is not such aproblem. Description will be made below regarding this point.

First, noise N in the case of applying Expression (5) can be obtained bygeometric-mean computing of dispersion 6 as shown in the followingExpression (17), assuming that there is no correlation between A colorcomponents (here, A color is equivalent to white), R color components, Gcolor components, and B color components.

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 17} \rbrack & \; \\ \begin{matrix}{N_{R}^{*} = \sqrt{N_{R}^{2} + {\alpha_{R}^{2}( {{\gamma_{A}^{2}N_{A}^{2}} + {\gamma_{R}^{2}N_{R}^{2}} + {\gamma_{G}^{2}N_{G}^{2}} + {\gamma_{B}^{2}N_{B}^{2}}} )}}} \\{N_{G}^{*} = \sqrt{N_{G}^{2} + {\alpha_{G}^{2}( {{\gamma_{A}^{2}N_{A}^{2}} + {\gamma_{R}^{2}N_{R}^{2}} + {\gamma_{G}^{2}N_{G}^{2}} + {\gamma_{B}^{2}N_{B}^{2}}} )}}} \\{N_{B}^{*} = \sqrt{N_{B}^{2} + {\alpha_{B}^{2}( {{\gamma_{A}^{2}N_{A}^{2}} + {\gamma_{R}^{2}N_{R}^{2}} + {\gamma_{G}^{2}N_{G}^{2}} + {\gamma_{B}^{2}N_{B}^{2}}} )}}}\end{matrix} \} & (17)\end{matrix}$

Also, in the event of applying Expression (12), when the respectivecoefficients ω and η are such as the above Expression (16-2), theinfrared light components IR and the blue components B are such as thefollowing Expression (18).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 18} \rbrack & \; \\ \begin{matrix}{{IR} = {{\gamma_{a} \times A} - ( {{\gamma_{r} \times R} + {\gamma_{g} \times G} + {\gamma_{b} \times B}} )}} \\\begin{matrix}{B^{*} = {B - {\alpha_{b} \times {IR}} + {\omega \times ( {B - \eta} ) \times {IR}}}} \\{= {{( {1 + {\alpha_{b}\gamma_{b}} + {\eta \; \omega \; \gamma_{b}}} ) \times B} +}} \\{{{( {{\alpha_{b}\gamma_{r}} + {\eta \; \omega \; \gamma_{r}}} ) \times R} +}} \\{{{( {{\alpha_{b}\gamma_{g}} + {\eta \; \omega \; \gamma_{g}}} ) \times G} +}} \\{{{( {{\alpha_{b}\gamma_{a}} - {\eta \; \omega \; \gamma_{a}}} ) \times A} +}} \\{{{\omega \; \gamma_{a} \times {AB}} - {\omega \; \gamma_{r} \times {RB}} - {\omega \; \gamma_{g} \times {GB}} - {\omega \; \gamma_{b} \times B^{2}}}}\end{matrix}\end{matrix} \} & (18)\end{matrix}$

Accordingly, for example, with regard to the blue components B, thefollowing Expression (19) is obtained, and noise NB* regarding the bluecomponents B can be obtained by taking a root at the left side ofExpression (19). Note that there are relations of σb1=NB*, σa=NA, σr=NR,σg=NG, and σb=NB between Expression (19) and Expression (17).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 19} \rbrack & \; \\ \begin{matrix}{\sigma_{b\; 1}^{2} = {{( {1 + {\alpha_{b}\gamma_{b}} + {\eta \; \omega \; \gamma_{b}}} )^{2} \times \sigma_{b}^{2}} + {( {{\alpha_{b}\gamma_{r}} + {\eta \; \omega \; \gamma_{r}}} )^{2} \times \sigma_{r}^{2}} +}} \\{{{( {{\alpha_{b}\gamma_{g}} + {\eta \; \omega \; \gamma_{g}}} )^{2} \times \sigma_{g}^{2}} + {( {{\alpha_{b}\gamma_{a}} - {\eta \; \omega \; \gamma_{a}}} )^{2} \times \sigma_{a}^{2}} +}} \\{{{( {\omega \; \gamma_{a}} )^{2} \times \sigma_{a}^{2}\sigma_{b}^{2}} + {( {\omega \; \gamma_{r}} )^{2} \times \sigma_{r}^{2}\sigma_{b}^{2}} + {( {\omega \; \gamma_{g}} )^{2} \times}}} \\{{{\sigma_{g}^{2}\sigma_{b}^{2}} + {( {\omega \; \gamma_{b}} )^{2} \times \sigma_{b}^{4}}}\;}\end{matrix} \} & (19)\end{matrix}$

FIGS. 101 and 102 are charts summarizing the estimated values and actualvalues of noise regarding the halogen light source (color temperature of3000 K) and fluorescent light. As can be understood from FIGS. 101 and102, in the event of subjecting the IR leakage light to correction usingnonlinear terms by applying Expression (12) as to the case of employingthe ordinary IR cut filter C5000, with the halogen light source (colortemperature of 3000 K), −0.530 dB (estimation), and −0.784 dB (actualmeasurement), and with the fluorescent light +2.854 dB (estimation), and+0.383 dB (actual measurement). As a result, we have found that evenunder any light source, noise deterioration is not a big problem.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

1. An apparatus with a substrate comprising: at least one visible lightdetection unit coupled to an image signal processing unit, each saidvisible light detection unit comprising a color filter adapted totransmit a corresponding wavelength region of visible light; at leastone infrared light detection unit adapted to detect a wavelength regionof infrared light and coupled to said image signal processing unit; andelectrical isolation between each light detection unit and the infraredlight detection unit, wherein, the infrared light detection unit ispositioned deeper within the substrate than each light detection unitrelative to a side on which light is incident on the device.
 2. Theapparatus of claim 1, wherein the signal processing unit corrects afirst signal received from the at least one visible light detection unitby subtracting a product from said first signal, said product resultingfrom multiplication of a second signal received from the at least oneinfrared light detection unit and a predetermined coefficient factor. 3.The apparatus according to claim 1, wherein said visible light detectionunit has no means for infrared light filtering.
 4. The apparatusaccording to claim 1, wherein the signal processing unit corrects thefirst signal by adding nonlinear signal components multiplied by thepredetermined coefficient.
 5. The apparatus according to claim 1,wherein the infrared light detection unit detects a unit signal of thewavelength region of infrared light and excludes the wavelength regionof visible light.
 6. The apparatus according to claim 5 wherein thesignal processing unit integrates the unit signal of the wavelengthregion of infrared light multiple times, and corrects a unit signaldetected by the visible light detection unit using the integrated unitsignal of the wavelength region of infrared light.
 7. The apparatusaccording to claim 1, further comprising a driving unit for controllingthe detection time of the infrared detection unit.
 8. The apparatus ofclaim 1, wherein the apparatus comprises a semiconductor device, thesemiconductor device comprising having, on a common substrate, the atleast one visible light detection unit and the at least one infraredlight detection unit.
 9. The apparatus according to claim 1, wherein thefirst detection unit and the second detection unit are disposedperiodically with a certain numerical ratio.
 10. The apparatus accordingto claim 8, wherein the first detection unit and the second detectionunit are disposed one to one.